Image compression based on tiled wavelet-like transform using edge and non-edge filters

ABSTRACT

An image processing system tiles an image data array. Each tile of image data is processed by applying a family of transform layers to the tile of image data so as to generate successive sets of transform coefficients. Each set of transform coefficients include edge coefficients positioned at outside boundaries of the set of transform coefficients and non-edge coefficients positioned at interior locations of the set of transform coefficients. The transform filters include a short edge transform filter that is applied to image data at boundaries of the tile and to coefficients positioned at and near boundaries of each of the earlier sets of transform coefficients so as to generate the edge coefficients, and a long interior filter that is applied to image data at interior locations of the tile and to coefficients at interior locations of the earlier sets of transform coefficients so as to generate the non-edge coefficients.

The present invention relates generally to the processing, compression,communication and storage of images in computer systems, personaldigital assistants, digital cameras and other devices, and particularlyto an image management system and method in which digitally encodedimages can be viewed in any specified window size and at a number ofresolutions, and can be printed, cropped, and otherwise manipulated.

BACKGROUND OF THE INVENTION

An image may be stored at a number of resolution levels. The encodedimage data for a lower resolution level is smaller, and thus takes lessbandwidth to communicate and less memory to store than the data for ahigher resolution level. When an image is stored for multi-resolutionuse, it would be desirable for the image data to be segregated into anordered group of sets or subfiles, where each additional subfileprovides the additional data needed to increase the resolution of theimage from one level to the next. Further, it would be desirable for thequantity of image data in each subfile, for increasing the imageresolution by a particular factor (such as 4), to be approximatelyproportional to the associated increase in resolution. For instance, ifeach resolution level differs from its neighboring resolution levels bya factor of 4 (e.g., level 0: 32×32, level 1: 64×64, level 2: 128×128,and so on), then the quantity of encoded image data for each resolutionlevel should be approximately 25% as much as the quantity of encodedimage data for the next higher resolution level. From another viewpoint,the quantity of data in the subfile(s) used to increase the imageresolution from a first level to the next should, ideally, beapproximately three times as much as the quantity of data in thesubfile(s) for the first level.

It is well known that wavelet compression of images automaticallygenerates several resolution levels. In particular, if N “layers” ofwavelet transforms are applied to an image, then N+1 resolution levelsof data are generated, with the last LL subband of data comprising thelowest resolution level and all the subbands of data together formingthe highest resolution level. For convenience, the “layers” of wavelettransforms will sometimes be called “levels”. Each of these resolutionlevels differs from its neighbors by a factor of two in each spatialdimension. We may label these resolution levels as Level 0 for thelowest, thumbnail level to Level N for the highest resolution level,which is the resolution of the final or base image.

A first aspect of the present invention is based on two observations.The first such observation is that, when using conventional as well asmost proprietary data compression and encoding methods, the quantity ofdata in the N levels generated by wavelet compression tends to decreasein a geometric progression. For instance, the quantity of data forresolution Level 0 is typically about 80% of the quantity of data forresolution Level 1, whereas ideally it should about 25% of the quantityof data for resolution Level 1. As a result, the data for Level 0contains significantly more data than is needed to display the Level 0image. Alternately stated, the data for Level 0 gives unnecessarily highquality for the low resolution display at Level 0, and therefore givesless compression than could potentially be obtained by providing onlythe information needed for displaying the image at the Level 0resolution level.

The second observation is that the low resolution image datacoefficients are quantized for full resolution display, not for lowresolution display, because these data coefficients are used not onlyfor generating a low resolution representation of the image, but arealso used when generating the higher resolution representations of theimage.

In accordance with this first aspect of the present invention, asalready indicated above, it would be desirable for the quantity of imagedata in the subarray or subfile for each resolution level to beapproximately proportional to the increase in resolution associated withthat resolution level.

A second aspect of the present invention is based on the observationthat wavelet transforms are conventionally applied across tile or blockboundaries of an image to avoid tile or block boundary artifacts in theregenerated image. A wavelet transform may be implemented as a FIR(finite impulse response) filter having an associated length. The“length” indicates the number of data samples that are used to generateeach coefficient. Wavelet transforms are generally symmetric about theircenter, and when the filter that implements the wavelet transform is atthe edge of a tile or block, typically half or almost half of the filterwill extend into a neighboring block or tile. As a result it is usuallynecessary to keep not only part of the neighboring tiles in memory whilewavelet transforming a tile of an image, it also necessary to keep inmemory the edge coefficients of the neighboring tiles for each level ofthe wavelet transform. Thus, avoiding tiling effects (also called tileborder effects or artifacts or edge artifacts) typically increases thememory requirements of the computer or device performing the wavelettransforms on an image, and may also increase the complexity of thetransform procedure because of the need to keep track of the memorylocations of edge data and coefficients from the neighboring tiles orblocks. In accordance with the second aspect of the present invention,it would be highly desirable to have a wavelet or wavelet-like transformthat can be applied to just the data for the image block beingprocessed, without having to also apply the transform to data fromneighboring blocks, and without creating noticeable edge artifacts.Having such a transform would decrease memory requirements and mightsimplify the wavelet compression of images.

It is well known in the prior art that digital images can be processed aportion at a time, instead of all at once, thereby reducing memoryrequirements. For instance, the DCT transform used for JPEG compressionand encoding of images is traditionally used on tiles of 8×8 pixels.However, a well known problem with tiling an image for processing isthat the tiling produces undesirable tile border effects. The bordereffects of DCT tiling in JPEG images are considered to be acceptablebecause the very small size of the tiles makes the tiling effectrelatively unnoticeable to the human eye.

However, using very small tiles such as 8×8 pixels is not practical whenusing wavelet or wavelet-like transforms in place of the DCT transform.Wavelet-like transforms have been shown to provide significantly betterdata compression than the DCT transform, and therefore usingwavelet-like transforms would be desirable if the tiling effect can beavoided while using a moderate amount of working memory.

It would therefore be desirable to provide an image processing systemand method that process images using a moderate amount of workingmemory, such as 8 to 20 KB, by transforming the image data using awavelet-like transform with moderately sized tiles, such as tiles of64×64, or 32×32, or 64×32 pixels, while at the same time avoiding thegeneration of undesirable tiling (tile border) effects.

A third aspect of the present invention is based on the observation thatthe optimal quantization level to be applied to wavelet coefficients notonly varies from one transform subband to another, but also varies fromone region of an image to another. In particular, regions of an imagethat contain many “features” (typically characterized by horizontal orvertical lines or edges) are harder to compress than regions with fewerfeatures. That is, such densely featured image regions cannot becompressed as much as less densely featured regions without causingdegradation in the quality of the image regions regenerated from thecompressed data. It would therefore be desirable to provide an imagecompression and encoding system with a quantization procedure that usessmaller quantization divisors to quantize the wavelet coefficients ofheavily featured regions than the quantization divisors used to quantizethe wavelet coefficients of regions having fewer features.

SUMMARY OF THE INVENTION

In summary, the present invention is an image processing system andmethod for applying a family of predefined transforms, such aswavelet-like transforms, to the image data for an image so as togenerate transform image data and for applying a data compression methodto the transform image data so as to generation image file. The imageprocessing system and method tiles a captured image, processing thetiles in a predefined order. The tiles are nonoverlapping portions ofthe image data. Each tile of image data is processed by applying apredefined family of transform layers to the tile of image data so as togenerate successive sets of transform coefficients. In a preferredembodiment, the transform layers are successive applications of a familyof wavelet-like decomposition transforms, including edge filters appliedto data at the boundaries of the data arrays being processed andinterior filters applied to data in the interior regions of the dataarrays.

The set of transform coefficients processed by each transform layerinclude edge coefficients positioned at outside boundaries of the set oftransform coefficients and non-edge coefficients positioned at interiorlocations of the set of transform coefficients. The sets of transformcoefficients include a last set of transform coefficients, produced bythe last transform layer, and one or more earlier sets of transformcoefficients.

The transform filters used include one or more edge transform filtersapplied to image data at boundaries of the tile and to coefficientspositioned at and near boundaries of each of the earlier sets oftransform coefficients so as to generate the edge coefficients, and oneor more interior filters applied to image data at interior locations ofthe tile and to coefficients at interior locations of the earlier setsof transform coefficients so as to generate the non-edge coefficients.The edge transform filters have shorter filter supports than theinterior transform filters, and both the edge transform filters and thelonger interior transform filters are applied only to image data withinthe tile and only to transform coefficients within the earlier sets oftransform.

The edge filters include a short, low spatial frequency filter thatweights the image datum closest to the boundary of the tile and thetransform coefficient closest to the boundary of each earlier set oftransform coefficients so as to as enable regeneration of the image fromthe transform coefficients without tile boundary artifacts.

At least some of the transform filters are preferably asymmetricboundary filters, extending to a first extent toward each tile'sboundary, and extending to a second, longer extent in a direction awayfrom the tile's boundary, but not extending over the tile's boundary.

In a preferred embodiment, the interior transform filters include acenter filter, for generating two to four high pass and two to four lowpass coefficients at or near the center of the data array beingprocessed. The center filter acts as a filter switch. Two distinct formsof the interior filter are used on alternate sides of the center filter.For instance, the interior filter may be centered on even data positionson one side of the center filter and centered on odd data positions onthe other side of the center filter.

The image processing system and method may also include imagereconstruction circuitry or procedures for successively applying a datadecompression method and an inverse transform to the image file so as togenerate a reconstructed image suitable for display on an image viewer.

In a second aspect of the present invention, the sets of transformcoefficients correspond to spatial frequency subbands of the image. Thesubbands are grouped in accordance with the transform layer thatgenerated them. For one or more respective groups of subbands, for eachtile of the image, one or more parameters are generated whose value isindicative of the density of image features in the tile. Each tile ofthe image is classified into one of a predefined set of categories inaccordance with the values of the one or more parameters. Based on theclassification of the tile, a set of quantization factors for the tileare selected, and then the transform coefficients of the tile are scaledby the selected set of quantization factors to as to generate a set ofquantized transform coefficients for the tile.

In a third aspect of the present invention the quantized transformcoefficients are encoded. While the coefficients for each group ofspatial frequency subbands are being encoded, a most significant set ofbit planes of those coefficients are stored in a first bitstream and theremaining least significant set of bit planes of the coefficients arestored in a second bitstream. From another viewpoint, the portions ofthe encoded coefficients (for a group of subbands) whose value exceeds apredefined threshold are stored in a first bitstream while the remainingportion of the encoded coefficients are stored in a second bitstream.When reconstructing an image from the image file at a specifiedresolution level, only the bitstreams corresponding to the specifiedresolution level are decoded and used to reconstruct the image. For someresolution levels, one or more of the bitstreams not used will containthe least significant portions (i.e., bit planes) of subbands whose moresignificant portions are contained in the bitstreams used to reconstructthe image at that resolution level.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional objects and features of the invention will be more readilyapparent from the following detailed description and appended claimswhen taken in conjunction with the drawings, in which:

FIG. 1 is a block diagram of a distributed computer system, including aweb server and a number of client computers, for distributingmulti-resolution images to the client computers.

FIG. 2 is a block diagram of a computer system in accordance with anembodiment of the present invention.

FIG. 3A schematically depicts the process of transforming a raw imageinto a transform image array and compressing the transform image arrayinto a compressed image file. FIG. 3B depicts a mapping of spatialfrequency subbands to NQS subbands used for encoding transformcoefficients.

FIG. 4 is a conceptual representation of the encoded data thatrepresents an image, organized to facilitate multi-resolutionregeneration of the image (i.e., at multiple resolution levels).

FIGS. 5A, 5B, 5C, 5D and 5E depict image storage data structures.

FIG. 6 is a high level flow chart of an image processing process towhich the present invention can be applied.

FIGS. 7A, 7B and 7C graphically depict a forward and inversewavelet-like data transformation procedure.

FIG. 8 depicts the spatial frequency subbands of wavelet coefficientsgenerated by applying multiple layers of a decomposition wavelet orwavelet-like transform to an array of image data.

FIG. 9 depicts a flow chart of a block classification method forselecting a set of quantization divisors for a block of an image.

FIGS. 10A and 10B depict a flow chart of a procedure for encoding thetransform coefficients for a block of an image.

FIGS. 11A, 11B and 11C depict a method of encoding values, calledMaxbitDepth values in a preferred embodiment, which represent the numberof bits required to encode the transform coefficients in each block andsubblock of an encoded image.

FIG. 12 is a high level flow chart of a compressed image reconstructionprocess to which the resent invention can be applied.

FIGS. 13A and 13B depict a flow chart of a procedure for decoding thetransform coefficients for an image and for reconstructing an image fromthe coefficients.

FIG. 14 is a block diagram of a digital camera in which one or moreaspects of the present invention are implemented.

FIG. 15 is a conceptual flow chart of a client computer downloading athumbnail image, then zooming in on the image, and then panning to a newpart of the image.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In this document, the terms “wavelet” and “wavelet-like” are usedinterchangeably. Wavelet-like transforms generally have spatialfrequence characteristics similar to those of conventional wavelettransforms, and are losslessly reversible, but have shorter filters thatare more computationally efficient.

The present invention may be implemented in a variety of devices thatprocess images, including a variety of computer systems, ranging fromhigh end workstations and servers to low end client computers, as wellas in application specific dedicated devices, such as digital cameras.

System for Encoding and Distributing Multi-Resolution Images

FIG. 1 shows a distributed computer system, including a web server 140and a number of client computers 120, for distributing multi-resolutionimages 190 to the client computers via a global communications network110, such as the Internet, or any other appropriate communicationsnetwork, such as a local area network or Intranet. An imaging encodingworkstation 150 prepares multi-resolution image files for distributionby the web server. In some embodiments, the web server 140 may alsoperform the image encoding tasks of the image encoding workstation 150.

A typical client device 120 will be a personal digital assistant,personal computer, workstation, or a computer controlled devicededicated to a particular task. The client device 120 will preferablyinclude a central processing unit 122, memory 124 (including high speedrandom access memory, and non-volatile memory such as disk storage), anda network interface or other communications interface 128 for connectingthe client device to the web server via the communications network 110.The memory 124 will typically store an operating system 132, a browserapplication or other image viewing application 134, an image decodermodule 180, and multi-resolution image files 190 encoded in accordancewith the present invention. In a preferred embodiment, the browserapplication 134 includes or is coupled to a Java™ (trademark of SunMicrosystems, Inc.) virtual machine for executing Java languageprograms, and the image decoder module is implemented as a Java™ appletthat is dynamically downloaded to the client device along with the imagefiles 190, thereby enabling the browser to decode the image files forviewing.

The web server 140 will preferably include a central processing unit142, memory 144 (including high speed random access memory, andnon-volatile memory such as disk storage), and a network interface orother communications interface 148 for connecting the web server toclient devices and to the image encoding workstation 150 via thecommunications network 110. The memory 144 will typically store an httpserver module 146 for responding to http requests, including request formulti-resolution image files 190.

The web server 140 may optionally include an image processing module 168with encoding procedures 172 for encoding images as multi-resolutionimages.

Computer System

Referring to FIG. 2, the image processing workstation 150 may beimplemented using a programmed general-purpose computer system. Thisfigure may also represent the web server, when the web server performsimage processing tasks. The computer system 150 may include:

one or more data processing units (CPU's) 152;

memory 154, which will typically include both high speed random accessmemory as well as non-volatile memory;

a user interface 156, including a display device 157 such as a CRT orLCD type display;

a network or other communication interface 158 for communicating withother computers as well as other devices;

a data port 160, such as for sending and receiving images to and from adigital camera (although such image transfers might also be accomplishedvia the network interface 158); and

one or more communication busses 161 for interconnecting the CPU(s) 152,memory 154, user interface 156, network interface 158 and data port 160.

The computer system's memory 154 stores procedures and data, typicallyincluding:

an operating system 162 for providing basic system services;

a file system 164, which may be part of the operating system;

application programs 166, such as user level programs for viewing andmanipulating images,

an image processing module 168, for performing various image processingfunctions including those that are the subject of the present document;

image files 190 representing various images; and

temporary image data arrays 192 for intermediate results generatedduring image processing and image regeneration.

The computer 150 may also include a http server module 146 (FIG. 1) whenthis computer 150 is used both for image processing and distribution ofmulti-resolution images.

The image processing module 168 may include an image encoder module 170,and an image decoder module 180. The image encoder module 170 producesmulti-resolution image files 190, the details of which will be discussedbelow. The image encoder module 170 may include:

an encoder control program 172, which controls the process ofcompressing and encoding an image (starting with a raw image array 189,which in turn may be derived from the decoding of an image in anotherimage file format);

a set of wavelet-like transform procedures 174 for applying wavelet-likefilters to image data representing an image;

a block classifier procedure 176 for determining the quantizationdivisors to be applied to each block (or band) of transform coefficientsfor an image;

a quantizer procedure 178 for quantizing the transform coefficients foran image; and

a sparse data encoding procedure 179, also known as an entropy encodingprocedure, for encoding the quantized transform coefficients generatedby the quantizer 178.

The procedures in the image processing module 168 store partiallytransformed images and other temporary data in a set of temporary dataarrays 192.

The image decoder module 180 may include:

a decoder control program 182 for controlling the process of decoding animage file (or portions of the image file) and regenerating the imagerepresented by the data in the image file;

a sparse data decoding procedure 184 for decoding the encoded, quantizedtransform coefficients stored in an image file into a correspondingarray of quantized. transform coefficients;

a de-quantizer procedure 186 for dequantizing a set of transformcoefficients representing a tile of an image; and

a set of wavelet-like inverse transform procedures 188 for applyingwavelet-like inverse filters to a set of dequantized transformcoefficients, representing a tile of an image, so as to regenerate thattile of the image.

Overview of Image Capture and Processing

Referring to FIG. 3, raw image data 200, obtained from a digitalcamera's image capture mechanism (FIG. 14) or from an image scanner orother device, is processed by “tiling the image data.” Morespecifically, the raw image is treated as an array of tiles 202, eachtile having a predefined size such as 64×64 (i.e., 64 rows by 64columns). In other embodiments, other tile sizes, such as 32×32 or 16×32or 128×128 or 64×128 may be used. The tiles are nonoverlapping portionsof the image data. A sufficient number of tiles are used to cover theentire raw image that is to be processed, even if some of the tilesoverhang the edges of the raw image. The overhanging portions of thetiles are filled with copies of boundary data values during the wavelettransform process, or alternately are filled with null data. Tilepositions are specified with respect to an origin at the upper leftcorner of the image, with the first coordinate indicating the Y positionof the tile (or a pixel or coefficient within the tile) and the secondcoordinate indicating the X position of the tile (or a pixel orcoefficient within the tile). Thus a tile at position 0,128 is locatedat the top of the image, and has its origin at the 128^(th) pixel of thetop row of pixels.

A wavelet or wavelet-like decomposition transform is successivelyapplied to each tile of the image to convert the raw image data in thetile into a set of transform coefficients. When the wavelet-likedecomposition transform is a one dimensional transform that is beingapplied to a two dimensional array of image data, the transform isapplied to the image data first in one direction (e.g., the horizontaldirection) to produce an intermediate set of coefficients, and then thetransform is applied in the other direction (e.g., the verticaldirection) to the intermediate set of coefficients so as to produce afinal set of coefficients. The final set of coefficients are the resultof applying the wavelet-like decomposition transform to the image datain both the horizontal and vertical dimensions.

The tiles are processed in a predetermined raster scan order. Forexample, the tiles in a top row are processed going from one end (e.g.,the left end) to the opposite end (e.g., the right end), beforeprocessing the next row of tiles immediately below it, and continuinguntil the bottom row of tiles of the raw image data has been processed.

The transform coefficients for each tile are generated by successiveapplications of a wavelet-like decomposition transform. A firstapplication of the wavelet decomposition transform to an initial twodimensional array of raw image data generates four sets of coefficients,labeled LL, HL1, LH1 and HH1. Each succeeding application of the waveletdecomposition transform is applied only to the LL set of coefficientsgenerated by the previous wavelet transformation step and generates fournew sets of coefficients, labeled LL, HLx, LHx and HHx, where xrepresents the wavelet transform “layer” or iteration. After the lastwavelet decomposition transform iteration only one LL set remains. Thetotal number of coefficients generated is equal to the number of datasamples in the original data array. The different sets of coefficientsgenerated by each transform iteration are sometimes called layers. Thenumber of wavelet transform layers generated for an image is typically afunction of the resolution of the initial image. For tiles of size64×64, or 32×32, performing five wavelet transformation layers istypical, producing 16 spatial frequency subbands of data: LL₅, HL₅, LH₅,HH₅, HL₄, LH₄, HH₄, HL₃, LH₃, HH₃, HL₂, LH₂, HH₂, HL₁, LH₁, HH₁. Thenumber of transform layers may vary from one implementation to another,depending on both the size of the tiles used and the amount ofcomputational resources available. For larger tiles, additionaltransform layers would likely be used, thereby creating additionalsubbands of data. Performing more transform layers will often producebetter data compression, at the cost of additional computation time, butmay also produce additional tile edge artifacts.

The spatial frequency subbands are grouped as follows. Subband group 0corresponds to the LL subband, where N is the number of transform layersapplied to the image (or image tile). Each other subband group icontains three subbands, LH_(i), HL_(i) and HH_(i). As will be describedin detail below, when the transform coefficients for a tile are encoded,the coefficients from each group of subbands are encoded separately fromthe coefficients of the other groups of subbands. In a preferredembodiment, a pair of bitstreams is generated to represent thecoefficients in each group of subbands. One of the bitstreams representsthe most significant bit planes of the coefficients in the group ofsubbands while the second bitstream represents the remaining, leastsignificant bit planes of the coefficients for the group of subbands.

The wavelet coefficients produced by application of the wavelet-liketransform are preferably quantized (by quantizer 178) by dividing thecoefficients in each subband of the transformed tile by a respectivequantization value (also called the quantization divisor). In thepreferred embodiment, a separate quantization divisor is assigned toeach subband. More particularly, as will be discussed in more detailbelow, a block classifier 176 generates one or more valuesrepresentative of the density of features in each tile of the image, andbased on those one or more values, a table of quantization divisors isselected for quantizing the coefficients in the various subbands of thetile.

The quantized coefficients produced by the quantizer 178 are encoded bya sparse data encoder 179 to produce a set of encoded subimage subfiles210 for each tile of the image.

Details of the wavelet-like transforms used in a preferred embodimentare described in detail below. Circuitry for performing the wavelet-liketransform of the preferred embodiment is very similar to the wavelettransform and data quantization methods described in U.S. Pat. No.5,909,518, “System and Method for Performing Wavelet and Inverse WaveletLike Transformations of Digital Data Using Only Add and Bit ShiftArithmetic Operations,” which is hereby incorporated by reference asbackground information.

The sparse data encoding method of the preferred embodiment is calledNested Quadratic Splitting (NQS), and is described in detail below. Thissparse data encoding method is an improved version of the NQS sparsedata encoding method described in U.S. Pat. No. 5,949,911, entitled“System and Method for Scalable Coding of Sparse Data Sets,” which ishereby incorporated by reference as background information.

FIG. 3B depicts a mapping of spatial frequency subbands to NQS subbandsused for encoding transform coefficients. In particular, in onepreferred embodiment seven spatial frequency subbands (LL₅, HL₅, LH₅,HH₅, HL₄, LH₄, and HH₄) are mapped to a single NQS subband (subband 0)for purposes of encoding the coefficients in these subbands. In otherwords, the coefficients in these seven spatial frequency subbands aretreated as a single top level block for purposes of NQS encoding. In onepreferred embodiment, NQS subbands 0, 1, 2 and 3 are encoded as four toplevel NQS blocks, the most significant bit planes of which are stored ina bitstream representing a lowest resolution level of the image inquestion.

Image Resolution Levels and Subimages

Referring to FIG. 4, an image is stored at a number of resolution levels0 to N, typically with each resolution level differing from itsneighbors by a resolution factor of four. In other words, if the highestresolution representation (at resolution level N) of the image containsX amount of information, the second highest resolution levelrepresentation N−1 contains X/4 amount of information, the third highestresolution level representation contains X/16 amount of information, andso on. The number of resolution levels stored in an image file willdepend on the size of the highest resolution representation of the imageand the minimum acceptable resolution for the thumbnail image at thelowest resolution level. For instance, if the full or highest resolutionimage is a high definition picture having about 16 million pixels (e.g.,a 4096×4096 pixel image), it might be appropriate to have sevenresolution levels: 4096×4096, 2048×2048, 1024×1024, 512×512, 256×256,128×128, and 64×64.

However, as shown in FIG. 4, one feature or aspect of the presentinvention is that when a multi-resolution image has more than, say,three or four resolution levels, the image is encoded and stored inmultiple “base image” files, each of which contains the data for two tofour of the resolution levels. Alternately, all the base images may bestored in a single file, with each base image being stored in a distinctbase image subfile or subfile data structure within the image file.

Each base image file (or subfile) contains the data for reconstructing a“base image” and one to three subimages (lower resolution levels). Forinstance, in the example shown in FIG. 4, the image is stored in threefiles, with a first file storing the image at three resolution levels,including the highest definition level and two lower levels, a secondfile stores the image at three more resolution levels (the fourth, fifthand sixth highest resolution levels) and a third file stores the imageat the two lowest resolution levels, for a total of eight resolutionlevels. Generally, each successive file will be smaller than the nextlarger file by a factor of about 2^(2X) where X is the number ofresolution levels in the larger file. For instance, if the first filehas three resolution levels, the next file will typically be smaller bya factor of 64 (2⁶).

As a result, an image file representing a group of lower resolutionlevels will be much smaller, and thus much faster to transmit to aclient computer, than the image file containing the full resolutionimage data. For instance, a user of a client computer might initiallyreview a set of thumbnail images, at a lowest resolution level (e.g.,32×32 or 64×64), requiring the client computer to review only thesmallest of the three image files, which will typically contain about0.024% as much data as the highest resolution image file. When the userrequests to see the image at a higher resolution, the client computermay receive the second, somewhat larger image file, containing about 64times as much data as the lowest resolution image file. This second filemay contain three resolution levels (e.g., 512×512, 256×256, and128×128), which may be sufficient for the user's needs. In the event theuser needs even high resolution levels, the highest resolution file willbe sent. Depending on the context in which the system is used, thevendor of the images may charge additional fees for downloading eachsuccessively higher resolution image file.

It should be noted that many image files are not square, but rather arerectangular, and that the square image sizes used in the above examplesare not intended to in any way limit the scope of the invention. Whilethe basic unit of information that is processed by the image processingmodules is a tile, which is typically a 64×64 or 32×32 array of pixels,any particular image may include an arbitrarily sized array of suchtiles. Furthermore, the image need not be an even multiple of the tilesize, since the edge tiles can be truncated wherever appropriate.

The designation of a particular resolution level of an image as the“thumbnail” image may depend on the client device to which the image isbeing sent. For instance, the thumbnail sent to a personal digitalassistant or mobile telephone, which have very small displays, may bemuch smaller than (for example, one sixteenth the size of) the thumbnailthat is sent to a personal computer; and the thumbnail sent to a devicehaving a large, high definition screen may be much larger than thethumbnail sent to a personal computer having a display of ordinary sizeand definition. When an image is to be potentially used with a varietyof client devices, additional base images are generated for the image sothat each type of device can initially receive an appropriately sizedthumbnail image.

When an image is first requested by a client device, the client devicemay specify its window size in its request for a thumbnail image, or theserver may determine the size of the client device's viewing window byquerying the client device prior to downloading the thumbnail image datato the client device. As a result, each client device receives anminimum resolution thumbnail that is appropriately sized for thatdevice.

Image File Data Structures

Referring to FIGS. 5A through 5E, when all the tiles of an image havebeen transformed, compressed and encoded, the resulting encoded imagedata is stored as an image file 190. The image file 190 includes headerdata 194 and a sequence of base image data structures, sometimes calledbase image subfiles 196. Each base image subfile 196 typically includesthe data for displaying the image at two or more resolution levels.Furthermore, each base image supports a distinct range of resolutionlevels. The multiple base images and their respective subimages togetherprovide a full range of resolution levels for the image, as conceptuallyrepresented in FIG. 4. While the resolution levels supported by the baseimage levels are non-overlapping in the preferred embodiment, in analternate embodiment the resolution levels supported by one base imagemay overlap with the resolution levels supported by another base image(for the same initial full resolution image).

In the preferred embodiment, each image file 190 is an html file orsimilarly formatted web page that contains a link 198, such as an objecttag or applet tag, to an applet 199 (e.g., a Java™ applet) that isautomatically invoked when the file is downloaded to a client computer.The header 194 and a selected one of the base images 196 are used asdata input to the embedded applet 199, which decodes and renders theimage on the display of user's personal digital assistant or computer.The operation of the applet is transparent to the user, who simply seesthe image rendered on his/her computer display. Alternately, the appletmay present the user with a menu of options including the resolutionlevels available with the base image subfile or subfiles included in theimage file, additional base image subfiles that may be available fromthe server, as well as other options such as image cropping options. Inan alternate embodiment, the client workstations include an application,such as a browser plug-in application, for decoding and rendering imagesin the file format of the present invention. Further, each image file210 has an associated data type that corresponds to the plug-inapplication. The image file 210 is downloaded along with an html orsimilarly formatted web page that includes an embed tag or object tagthat points to the image file. As a result, when the web page isdownloaded to a client workstation, the plug-in application isautomatically invoked and executed by the client computer's. As aresult, the image file is decoded and rendered and the operation of theplug-in application is transparent to the user.

The image file 190-A shown in FIG. 5A represents one possible way ofstoring a multi-resolution image, and is particularly suitable forstoring a multi-resolution image in a server. In a client computer, theimage file 190-B as shown in FIG. 5B may contain only one base image196. In addition, the client version of the image file 190 may contain alink 201 to the image file 190-A in the server. The link 201 is used toenable a user of the client computer to download other base images (atother resolution levels) of the same image. Alternately, the link 201 isa Java™ (trademark of Sun Microsystems) script for requesting an imagefile containing any of the higher resolution base images from the webserver. If there is a charge for obtaining the higher resolution imagefile, the script will invoke the execution of the server procedure forobtaining payment from the requesting user.

In yet another alternate embodiment, a multi-resolution image may bestored in the server as a set of separate base image files 190-B, eachhaving the format shown in FIG. 5B. This has the advantage of providingimage files 190-B that are ready for downloading to client computerswithout modification.

Referring to FIG. 5A again, the header 194 of the image file includesthe information needed to access the various base image subfiles 196. Inparticular, in a preferred embodiment the header 194 stores:

an identifier or the URL of the image file in the server;

a parameter value that indicates the number of base image subfiles 196in the file (or the number of base image files in embodiments in whicheach base image is stored in a separate file);

the size of each base image data structure; and

an offset pointer to each base image data structure (or a pointer toeach base image file in embodiments in which each base image is storedin a separate file).

Each base image subfile 196 has a header 204 and a sequence ofbitstreams 206. The bitstreams are labeled 1 a, 1 b, to N, where N isthe number of resolution levels supported by the base image in question.The meaning of the labels “1 a” and the like will be explained below.The information in each bit stream 206 will be described in full detailbelow. The header data 204 of each base image subfile includes fieldsthat indicate:

the size of the base image subfile (i.e., the amount of storage occupiedby the base image subfile);

the size of the tiles (e.g., the number of rows and columns of pixels)used to tile the base image, where each tile is separately transformedand encoded, as described below;

the color channel components stored for this base image subfile;

the transform filters used to decompose the base image (e.g., differentsets of transform filters may be used on different images);

the number of spacial frequency subbands encoded for the base image(i.e., for each tile of the base image);

the number of resolution levels (also called subimages) supported by thebase image;

the number of bitstreams encoded for the base image (i.e., for each tileof the base image); and

information for each of the bitstreams.

The header information for each bitstream in the base image subfile mayinclude:

an offset pointer to the bitstream to indicate its position within theimage file (or within the base image subfile);

the size of bitstream (how much data is in the bitstream);

the range of spatial frequency subbands included in the bitstream;

the number of color channels in the bitstream;

the range of bit planes included in the bitstream, which indicates howthe bit planes of the coefficients in the subbands were divided betweensignificant, insignificant and possibly mid-significant portions; and

a table of offset pointers to the tiles 208 within the bitstream.

Each bitstream 206 includes a sequence of tile subarrays 208, each ofwhich contains the i^(th) bitstream for a respective tile of the image.The bitstream 206 may optionally include a header 209 having fields usedto override parameters specified for the base image by the base imageheader 204. When the image file contains a cropped image, the set oftile subarrays 208 included in the image file is limited to those neededto represent the cropped image.

In a preferred embodiment, the image file header 194 also includesparameters indicating “cropped image boundaries.” This is useful forpartial copies of the image file that contain data only for a croppedportion of the image, which in turn is very useful when a clientcomputer is being used to perform pan and zoom operations on an image.For instance, a user may have requested only a very small portion of theoverall image, but at very high resolution. In this case, only the tilesof the image needed to display the cropped portion of the image will beincluded in the version of the image file sent to the user's clientcomputer, and the cropped image boundary parameters are used to conveythis information to the procedures that render the image on the clientcomputer. Two types of image cropping information are provided by theimage file header 194: cropping that applies to the entire image file,and any further cropping that applies to specific subimages. Forinstance, when a client computer first receives an image, it mayreceived just the lowest resolution level subimage of a particular baseimage, and that subimage will typically not be cropped (compared to thefull image). When the client zooms in on a part of the image at aspecified higher resolution level, only the tiles of data needed togenerate the portion of the image to be viewed on the client computerare sent to the client computer, and thus new cropping parameters willbe added to the header of the image file stored (or cached) in theclient computer to indicate the cropping boundaries for the subimagelevel or levels downloaded to the client computer in response to theclient's image zoom command.

The table of offset pointers to tiles that is included in the base imageheader for each bitstream in the base image is also used during zoomingand panning. In particular, referring to FIG. 15, when an image file isfirst downloaded by a client computer or device (240), the higher levelbitstreams may be unpopulated, and thus the table of offset pointerswill initially contain null values. When the user of the client deviceszooms in on the image, the data for various tiles of the higher levelbitstreams are downloaded to the client device, as needed (242), and thetable of offset pointers to tiles is updated to reflect the tiles forwhich data have been downloaded to the client computer. When the clientfurther pans across the image at the zoomed or higher resolution level,additional tiles of information are sent to the client computer asneeded, and the cropping information in the image file header 194 andthe tile offset information in the base image header are again updatedto reflect the tiles of data stored for each bitstream (244).

Referring again to FIGS. 5A-5E, the information in the headers of theimage file and the base image subfiles enables quick indexing into anypart of the file, which enables a computer or other device to locate thebeginning or end of any portion of the image, at any resolution level,without having to decode the contents of any other portions of the imagefile 190. This is useful, for example, when truncating the image file190 so as to generate a lower image quality version of the file, or acropped image version of the file, such as for transmission over acommunications network to another computer or device.

In some of the discussions that follow, the terms “subimage” and“differential subimage” will be used with respect to the bitstreams 206as follows. Generally, any subimage of a base image will include all thebitstreams from bitstream 1 a through a particular last bitstream, suchas bitstream 3. This group of contiguous bitstreams constitute the dataneeded to reconstruct the image at a particular resolution level, hereincalled a subimage. A “differential subimage” consists of the additionalbitstreams needed to increase the image resolution from one subimagelevel to the next. For instance, bitstreams 1 c, 2 b and 3 mighttogether be called a differential subimage because these bitstreamscontain the data needed to double the resolution of the subimagegenerated from bitstreams 1 a through 2 a.

Referring to FIG. 5C, the encoded data 190-C representing a base imageis initially stored in “tile order.” The image file 190-C includes aheader 222 and a set of tile subfiles 220. Referring to FIG. 5D, eachtile subfile 220 contains a header 224 denoting the quantization tableused to encode the tile, offset pointers to the bitstreams within thesubfile, and other information. The tile subfile 220 for each tile alsocontains a set of bitstream subarrays 226. Each tile bitstream subarray226 contains encoded data representing either the most significant bitplanes, least significant bit planes or a middle set of bit planes of arespective set of NQS subbands (see FIG. 3B) of the tile. The followingtable shows an example of bit plan mappings to bitstream subarrays:

NQS Subband Nos. Resolution 0 to 3 4, 5, 6 7, 8, 9 16 × 16 S 32 × 32 S +MS S 64 × 64 S + MS + IS S + IS All

In this table, the bit planes corresponding to S, MS and IS differ foreach NQS subband. These bit plane ranges are specified in the header ofthe base image subfile. For instance, for NQS subbands 0 to 3, S maycorresponding to bit planes 16 to 7, MS may correspond to bit planes 6to 4, and IS may correspond to bit planes 3 to 0; while for NQS subbands4 to 6, S may corresponding to bit planes 16 to 5, and IS may correspondto bit planes 4 to 0.

Bitstreams 1 a, 1 b and 1 c contain the encoded data representing themost significant, middle and least significant bit planes of NQSsubbands 0, 1, 2 and 3, respectively. Bitstreams 2 a and 2 b contain theencoded data representing the most significant and least significant bitplanes, respectively, of NQS subbands 4, 5 and 6, which correspond tothe LH₂, HL₂ and HH₂ subbands. Bitstream 3 contains all the bit planesof the encoded data representing NQS subbands 7, 8 and 9, whichcorrespond to the LH₁, HL₁, and HH₁, subbands, respectively.

The tile subfiles 220 may be considered to be “temporary” files, becausethe encoded tile data is later reorganized from the file format of FIGS.5C and 5D into the file format shown in FIG. 5A.

FIG. 5E shows a specific example of a base image subfile 196, labeled196A. The base image subfile contains twelve bitstreams 206, which areused to generate the base image and two lower resolution subimages. Thebase image has been transformed with five layers of wavelet transforms,producing sixteen spatial frequency subbands of data, which have beenencoded and organized into three subimages, including the base image.The number of subimages is somewhat arbitrary, since the subbandsgenerated by five transform layers could be used to generate as many assix subimages. However, using this base image subfile to generate verysmall subimages is not efficient in terms of memory or storageutilization, and therefore it will often be preferred to use a smallerbase image subfile to generate smaller subimages.

In FIG. 5E, the base image has been processed by five transform layers,but the resulting data has been organized into just three subimagelevels instead of six. Effectively, the last three transform layers,which convert subband LL₂ into ten subbands (LL₅, LH₅, HL₅, HH₅, LH₄,HL₄, HH₄, LH₃, HL₃ and HH₃), are not used to generate an extra subimagelevel. Rather, the last three transform layers are used only to producebetter data compression.

As shown in FIG. 5E, when the five transform layers of image data aremapped to three subimages, the mapping of bitstream data subarrays 206to subimages is as follows:

subimage 0, the lowest level subimage, corresponds to bitstream subarray206-1 a, which contains the most significant bit planes of NQS subbands0 to 3 (see FIG. 3B);

subimage 1 corresponds to bitstreams 206-1 a, 206-1 b and 206-2 a; and

subimage 2, the base image, corresponds to all the bitstreams 206 in thebase image subfile.

When the transform layers are mapped to more subimages (subimage levels)than in the example shown in FIG. 5E, the first bitstream 206-1 a willinclude fewer of the spatial frequency subbands.

A sparse data encoding technique is used to encode the transformcoefficients for each group of subbands of each tile so that it takesvery little data to represent arrays of data that contain mostly zerovalues. Typically, higher frequency portions (i.e., subbands) of thetransformed, quantized image data will contain more zero values thannon-zero values, and further most of the non-zero values will haverelatively small absolute value. Therefore, the higher level bit planesof many tiles will be populated with very few non-zero bit values.

Tiled Wavelet Transform Method

Referring to FIG. 6, the process for generating an image file beginswhen an image is captured by the image capture device (step 250). If theimage size is variable, the size of the captured image is determined andthe number of rows and columns of tiles needed to cover the image datais determined (step 252). If the image size is always the same, step 252is not needed.

Next, all the tiles in the image are processed in a predetermined order,for example in raster scan order, by applying a wavelet-likedecomposition transform to them in both the horizontal and verticaldirections, then quantizing the resulting transform coefficients, andfinally by encoding the quantized transform coefficients using a sparsedata compression and encoding procedure (step 254). The encoded data foreach tile is stored in a temporary file or subfile, such as in theformat shown in FIG. 5D.

After all the tiles in the image have been processed, a multi-resolutionimage file containing all the encoded tiles is stored in non-volatilememory (step 256). More specifically, the encoded tile data from thetemporary files is written into an output bitstream file in resolutionreversed order, in the file format shown in FIG. 5A. “Resolutionreversed order” means that the image data is stored in the file with thelowest resolution bitstream first, followed by the next lowestresolution bitstream, and so on.

The wavelet-like decomposition transform used in step 254 is describedin more detail below, with reference to FIGS. 7A, 7B and 7C. Thequantization and sparse data encoding steps are also described in detailbelow.

After the initial image has been processed, encoded and stored as amulti-resolution image file, typically containing two to four resolutionlevels, if more than one base image is to be included in the image file(257), the original image is down-sampled and anti-aliased so as togenerate a new base image (258) that is smaller in each dimension by afactor of 2^(X) where X is the number of subimage levels in thepreviously generated multi-resolution image file. Thus, the new baseimage will be a factor of 4 smaller than the smallest, lowest-resolutionsubimage of the base image. The new base image is then processed in thesame way as the previous base image so as to generate an additional, butmuch smaller, encoded multi-resolution base image that is added to theimage file. If the original base image had sufficiently high resolution,a third base image may be formed by performing a second round ofdown-sampling and anti-aliasing, and a third encoded multi-resolutionbase image file may be stored in the image file. The last encoded baseimage may contain fewer subimage levels than the others, and in someembodiments may contain only a single resolution level, in which casethat image file is effectively a thumbnail image file.

In an alternate embodiment each encoded base image is stored in aseparate image file, and these image files are linked to each othereither by information stored in the headers of the image files, or byhtml (or html-like) links.

In a preferred embodiment, the down-sampling filter is a one-dimensionalFIR filter that is applied first to the rows of the image and then tothe columns, or vice versa. For example, if the image is to bedown-sampled by a factor of 4 in each dimension (for a factor of 16reduction in resolution), the FIR filter may have the following filtercoefficients:

Filter A=(−3 −3 −4 −4 10 10 29 29 29 29 10 10 −4 −4 −3 −3) 1/128.

This exemplary filter is applied to a set of 14 samples at a time toproduce one down-sampled value, and is then shifted by four samples andis then applied again. This repeats until L/4 down-sampled values havebeen generated, where L is the number of initial samples (i.e., pixelvalues). At the edges of the image data array, reflected data is usedfor the filter coefficients that extend past the edge of the image data.For instance at the left (or top) edge of the array, the first sixcoefficients are applied to reflected data values, the four “29/128”coefficients are applied to the first four pixel values in the row (orcolumn) being filtered, and the last six coefficients are applied to thenext six pixels in the row (or column).

If an image is to be down-sampled by a factor of 8, the above describedfilter is applied to down-sample by a factor of 4, and then a secondfilter is applied to further down-sample the image data by anotherfactor of 2. This second filter, in a preferred embodiment, is a FIRfilter that has the following filter coefficients:

Filter B=(−3 −4 10 29 29 10 −4 −3) 1/64.

Alternately, a longer filter could be used to achieve the down-samplingby a factor of 8 in one filter pass.

The down-sampling filters described above have the following properties:they are low-pass filters with cut-off frequencies at one quarter andone half the Nyquist frequency, respectively; each filter coefficient isdefined by a simple fraction in which the numerator is an integer andthe denominator is a positive integer power of 2 (i.e., a number of theform 2^(N), where N is a positive integer). As a result of these filterproperties, the down-sampling can be performed very efficiently whilepreserving the spatial frequency characteristics of the image andavoiding aliasing effects.

While the order in which the down-sampling filter(s) are applied to anarray of image data (i.e., rows and then columns, or vice versa) willaffect the specific down-sampled pixel values generated, the affect onthe pixel values is not significant. Other down-sampling filters may beused in alternate embodiments of the invention.

Wavelet-Like Decomposition Using Edge, Interior and Center TransformFilters

FIGS. 7A-7C schematically represent the process of performing awavelet-like decomposition on a set of image data X₀ to X_(2n−1) togenerate a set of coefficients L₀ to L_(n−1) and H₀ to H_(n−1) where theL coefficients represent the low spatial frequency components of theimage data and the H coefficients represent the high spatial frequencycomponents of the image data.

In the preferred embodiment, the wavelet-like transform that is appliedis actually two filters. A first filter, T1, called the edge filter, isused to generate the first two and last two coefficients in the row orcolumn of transform coefficients that are being generated, and a secondfilter T2, called the interior filter, is used to generate all the othercoefficients in the row or column of transform coefficients beinggenerated. The edge filter, T1, is a short filter that is used totransform data at the edges of a tile or block, while the interiorfilter T2 is a longer filter that is used to transform the data awayfrom the edges of the tile or block. Neither the edge filter nor theinterior filter uses data from outside the tile or block. As a result,the working memory required to apply the wavelet-like transform of thepresent invention to an array of image data is reduced compared to priorart systems. Similarly, the complexity of the circuitry and/or softwarefor implementing the wavelet-like transform of the present invention isreduced compared to prior art systems.

In the preferred embodiment, the edge filter includes a first, veryshort filter (whose “support” covers two to four data values) forgenerating the first and last coefficients, and a second filter forgenerating the second and second to last coefficients. The second edgefilter has a filter support that extends over three to six data values,and thus is somewhat longer than the first edge filter but shorter thanthe interior filter T2. The interior filter for generating the othercoefficients typically has a filter support of seven or more datavalues. The edge filter, especially the first edge filter for generatingthe first and last high spatial frequency coefficient values, isdesigned to minimize edge artifacts while not using any data fromneighboring tiles or blocks, at a cost of decreased data compression.Stated in another way, the edge filter of the present invention isdesigned to ensure accurate reproduction of the edge values of the dataarray being processed, which in turn minimizes edge artifacts when theimage represented by the data array is regenerated.

In the preferred embodiment, the wavelet-like decomposition transformapplied to a data array includes a layer 1 wavelet-like transform thatis distinct from the wavelet-like transform used when performing layers2 to N of the transform. In particular, the layer 1 wavelet-liketransform uses shorter filters, having shorter filter supports, than thefilters used for layers 2 to N. One of the reasons for using a differentwavelet-like transform (i.e., a set of transform filters) for layer 1 isto minimize boundary artifacts by using short filters for layer 1, whichtend to minimize boundary artifacts. Another reason for using adifferent wavelet-like transform (i.e., a set of transform filters) forlayer 1 than for the other layers is to minimize rounding errorsintroduced by the addition of a large number of scaled values. Roundingerrors, which occur primarily when filtering the raw image data duringthe layer 1 transform, can sometimes cause noticeable degradation in thequality of the image regenerated from the encoded image data.

The equations for the wavelet-like decomposition transform used in thepreferred embodiment are presented below.

Layer 1 Forward Wavelet-Like Transform

T1 and T2 Forward Transforms (Low Frequency):

Y _(k) =X _(2k) −X _(2k+1) k=0, 1, . . . ,n−1

${L_{k} = {{X_{{2k} + 1} + \left\lbrack \frac{Y_{k} + 1}{2} \right\rbrack} = {{\frac{X_{2k} + X_{{2k} + 1} + 1}{2}\quad k} = 0}}},1,\ldots \quad,{n - 1}$

T1 Forward Transform (Edge Filter - High Frequency):$H_{0} = {Y_{0} + \left\lbrack \frac{{- L_{0}} + L_{1} + 1}{2} \right\rbrack}$$H_{1} = {Y_{1} + \left\lbrack \frac{{- L_{0}} + L_{2} + 2}{4} \right\rbrack}$$H_{n - 2} = {Y_{n - 2} + \left\lbrack \frac{{- L_{n - 3}} + L_{n - 1} + 2}{4} \right\rbrack}$$H_{n - 1} = {Y_{n - 1} + \left\lbrack \frac{{- L_{n - 2}} + L_{n - 1} + 1}{2} \right\rbrack}$

T2 Forward Transform (Interior Filter - High Frequency):$H_{k} = {Y_{k} + \left\lbrack \frac{{3L_{k - 2}} - {22L_{k - 1}} + {22L_{k + 1}} - {3L_{k + 2}} + 32}{64} \right\rbrack}$k = 2, …  , n − 3

Layer 1 Inverse Wavelet-Like Transform

T1 Inverse Transform (Edge Filter - High Frequency):$Y_{0} = {H_{0} - \left\lbrack \frac{{- L_{0}} + L_{1} + 1}{2} \right\rbrack}$$Y_{1} = {H_{1} - \left\lbrack \frac{{- L_{0}} + L_{2} + 2}{4} \right\rbrack}$$Y_{n - 2} = {H_{n - 2} - \left\lbrack \frac{{- L_{n - 3}} + L_{n - 1} + 1}{4} \right\rbrack}$$Y_{n - 1} = {H_{n - 1} - \left\lbrack \frac{{- L_{n - 2}} + L_{n - 1} + 1}{2} \right\rbrack}$

T2 Inverse Transform (Interior Filter):$Y_{k} = {H_{k} - \left\lbrack \frac{{3L_{k - 2}} - {22L_{k - 1}} + {22L_{k + 1}} - {3L_{k + 2}} + 32}{64} \right\rbrack}$k = 2, …  , n − 3${X_{{2k} + 1} = {{L_{k} - {\left\lbrack \frac{Y_{k} + 1}{2} \right\rbrack \quad k}} = 0}},1,\ldots \quad,{n - 1}$

 X _(2k) =Y _(k) +X _(2k+1) k=0, 1, . . . , n−1

Forward Wavelet-Like Transform: Layers 2 to N

The equations for the preferred embodiment of the forward wavelet-likedecomposition transform for transform levels 2 through N (i.e., allexcept level 1) are shown next. Note that “2n” denotes the width of thedata, as measured in data samples, that is being processed by thetransform; “n” is assumed to be a positive integer. The edge filter T1is represented by the equations for H₀, H_(n−1), L₀, and L_(n−1), andhas a shorter filter support than the interior filter T2.

In an alternate embodiment, the same wavelet-like decompositiontransforms are used for all layers. For example, the wavelet-likedecomposition transform filters shown here for layers 2 to N would alsobe used for the layer 1 decomposition (i.e., for filtering the raw imagedata).$H_{0} = {X_{1} - {\left\lbrack \frac{X_{0} + X_{2} + 1}{2} \right\rbrack \quad \left( {{edge}\quad {filter}} \right)}}$${H_{k} = {X_{{2k} + 1} - \left\lbrack \frac{{9\left( {X_{2k} + X_{{2k} + 2}} \right)} - X_{{2k} - 2} - X_{{2k} + 4} + 8}{16} \right\rbrack}}\quad$${k = 1},\ldots \quad,{\frac{n}{2} - 3}$$H_{\frac{n}{2} - 2} = {X_{n - 3} - {\left\lbrack \frac{X_{n - 4} + X_{n - 2} + 1}{2} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$H_{\frac{n}{2} - 1} = {X_{n - 1} - {\left\lbrack \frac{{11X_{n - 2}} + {5X_{n + 1}} + 8}{16} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$H_{\frac{n}{2}} = {X_{n} - {\left\lbrack \frac{{5X_{n - 2}} + {11X_{n + 1}} + 8}{16} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$H_{\frac{n}{2} + 1} = {X_{n + 2} - {\left\lbrack \frac{X_{n + 1} + X_{n + 3} + 1}{2} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$${H_{k} = {X_{2k} - \left\lbrack \frac{{9\left( {X_{{2k} - 1} + X_{{2k} + 1}} \right)} - X_{{2k} - 3} - X_{{2k} + 3} + 8}{16} \right\rbrack}}\quad$$\quad {{k = {\frac{n}{2} + 2}},\ldots \quad,{n - 2}}$$H_{n - 1} = {X_{{2n} - 2} - {\left\lbrack \frac{X_{{2n} - 3} + X_{{2n} - 1} + 1}{2} \right\rbrack \quad \left( {{edge}\quad {filter}} \right)}}$$L_{0} = {{X_{0} + \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack} = {\frac{{7X_{0}} + {2X_{1}} - X_{2} + 3}{8}\quad \left( {{edge}{\quad \quad}{filter}} \right)}}$$L_{1} = {X_{2} + {\left\lbrack \frac{H_{0} + H_{1} + 2}{4} \right\rbrack \quad \left( {{edge}\quad {filter}} \right)}}$${L_{k} = {X_{2k} + \left\lbrack \frac{{5\left( {H_{k - 1} + H_{k}} \right)} - H_{k - 2} - H_{k + 1} + 8}{16} \right\rbrack}}\quad$${k = 1},\ldots \quad,{\frac{n}{2} - 3}$$L_{\frac{n}{2} - 2} = {X_{n - 4} + {\left\lbrack \frac{H_{\frac{n}{2} - 3} + H_{\frac{n}{2} - 2} + 2}{4} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$L_{\frac{n}{2} - 1} = {X_{n - 2} + {\left\lbrack \frac{{2H_{\frac{n}{2} - 2}} + {2H_{\frac{n}{2} - 1}} - H_{\frac{n}{2}} + 4}{8} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$L_{\frac{n}{2}} = {X_{n + 1} + {\left\lbrack \frac{{2H_{\frac{n}{2} + 1}} + {2H_{\frac{n}{2}}} - H_{\frac{n}{2} - 1} + 4}{8} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$$L_{\frac{n}{2} + 1} = {X_{n + 3} + {\left\lbrack \frac{H_{\frac{n}{2} + 1} + H_{\frac{n}{2} + 2} + 2}{4} \right\rbrack \quad \left( {{center}\quad {filter}} \right)}}$${L_{k} = {X_{{2k} + 1} + \left\lbrack \frac{{5\left( {H_{k} + H_{k + 1}} \right)} - H_{k - 1} - H_{k + 2} + 8}{16} \right\rbrack}}\quad$${k = {\frac{n}{2} + 2}},\ldots \quad,{n - 3}$$L_{n - 2} = {X_{{2n} - 3} + {\left\lbrack \frac{H_{n - 2} + H_{n - 1} + 2}{4} \right\rbrack \quad \left( {{edge}\quad {filter}} \right)}}$$\begin{matrix}{L_{n - 1} = {X_{{2n} - 1} + \left\lbrack \frac{H_{n - 1} + 2}{4} \right\rbrack}} \\{= {\frac{{7X_{{2n} - 1}} + {2X_{{2n} - 2}} - X_{{2n} - 3} + 3}{8}\quad \left( {{edge}{\quad \quad}{filter}} \right)}}\end{matrix}$

The general form of the decomposition transform equations, shown above,applies only when n is at least ten. When n is less than ten, some ofthe equations for terms between the edge and middle terms are droppedbecause the number of coefficients to be generated is too few to requireuse of those equations. For instance, when n=8, the two equations forgenerating L_(k) will be skipped.

Discussion of Attributes of Transform Filter

It is noted that the edge transform filter T1 for generating L₀ andL_(n−1) has a filter support of just three input samples at the edge ofthe input data array, and is weighted so that 70% of the value of thesecoefficients is attributable to the edge value X₀ or X_(2n−1) at thevery boundary of the array of data being filtered. The heavy weightingof the edge input datum (i.e., the sample closest to the array boundary)enables the image to be reconstructed from the transform coefficientssubstantially without tile boundary artifacts, despite the fact that theedge and interior filters are applied only to data within the tile whengenerating the transform coefficients for the tile. The layer 1 edgetransform filter T1 for generating L₀ and L_(n−1) is weighted so that50% of the value of these coefficients is attributable to the edge valueX_(2n−1) at the very boundary of the data array being filtered.

The interior transform filters in the preferred embodiment are notapplied in a uniform manner across the interior of the data array beingfiltered. Furthermore, the interior filter includes a center filter forgenerating four high pass and four low pass coefficients at or near thecenter of the data array being filtered. In alternate embodiments thecenter filter may generate as few as two high pass and two low passcoefficients. The center filter is used to transition between the leftand right (or upper and lower) portions of the interior filter. Thetransition between the two forms of the interior filter is herein called“filter switching.” One half of the interior filter, excluding thecenter filter, is centered on even numbered data or coefficientpositions while the other half of the interior filter is centered ondata at odd data positions. (The even and odd data positions of thearray are, of course, alternating data positions.) While the equationsas written place the center filter at the middle of the array, thecenter filter can be positioned anywhere within the interior of the dataarray, so long as there is a smooth transition between the edge filterand the interior filter. Of course, the inverse transform filter must bedefined so as to have an inverse center filter at the some position asthe forward transform filter.

Transform Equations for Small Data Arrays, for Layers 2 to N

When n is equal to four, the transform to be performed can berepresented as:

(X₀,X₁,X₂, X₃, X₄, X₅, X₆, X₇)→(L₀, L₁, L₂, L₃; H₀, H₁, H₂, H₃)

and the above general set of transform equations is reduced to thefollowing: $\begin{matrix}{H_{0} = {X_{1} - \left\lbrack \frac{X_{0} + X_{2} + 1}{2} \right\rbrack}} \\{H_{1} = {X_{3} - \left\lbrack \frac{{11X_{2}} + {5X_{5}} + 8}{16} \right\rbrack}} \\{H_{2} = {X_{4} - \left\lbrack \frac{{5X_{2}} + {11X_{5}} + 8}{16} \right\rbrack}} \\{H_{3} = {X_{6} - \left\lbrack \frac{X_{5} + X_{7} + 1}{2} \right\rbrack}} \\{L_{0} = {X_{0} + \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}} \\{L_{1} = {X_{2} + \left\lbrack \frac{{2H_{0}} + {2H_{1}} - H_{2} + 4}{8} \right\rbrack}} \\{L_{2} = {X_{5} + \left\lbrack \frac{{2H_{3}} + {2H_{2}} - H_{1} + 4}{8} \right\rbrack}} \\{L_{3} = {X_{7} + \left\lbrack \frac{H_{3} + 2}{4} \right\rbrack}}\end{matrix}$

When n is equal to two, the transform to be performed can be representedas:

(X₀, X₁, X₂, X₃)→(L₀, L₁; H₀, H₁)

and the above general set of transform equations is reduced to thefollowing: $\begin{matrix}{H_{0} = {X_{1} - \left\lbrack \frac{X_{0} + X_{3} + 1}{2} \right\rbrack}} \\{H_{1} = {X_{2} - \left\lbrack \frac{X_{0} + X_{3} + 1}{2} \right\rbrack}} \\{L_{0} = {X_{0} + \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}} \\{L_{1} = {X_{3} + \left\lbrack \frac{H_{1} + 2}{4} \right\rbrack}}\end{matrix}$

Inverse Wavelet-Like Transform: Layers 2 to N

The inverse wavelet-like transform for transform layers 2 through N(i.e., all except layer 1), used in the preferred embodiment, are shownnext. The general form of the transform equations applies only when n isat least ten. When n is less than ten, some of the equations for termsbetween the edge and middle terms are dropped because the number ofcoefficients to be generated is too few to require use of thoseequations.$X_{0} = {L_{0} - \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}$$X_{2} = {L_{1} - \left\lbrack \frac{H_{0} + H_{1} + 2}{4} \right\rbrack}$${X_{2k} = {L_{k} - \left\lbrack \frac{{5\left( {H_{k - 1} + H_{k}} \right)} - H_{k - 2} - H_{k + 1} + 8}{16} \right\rbrack}}\quad$${k = 2},\ldots \quad,{\frac{n}{2} - 3}$$X_{n - 4} = {L_{\frac{n}{2} - 2} - \left\lbrack \frac{H_{\frac{n}{2} - 3} + H_{\frac{n}{2} - 2} + 2}{4} \right\rbrack}$${X_{{2k} + 1} = {L_{k} - \left\lbrack \frac{{5\left( {H_{k} + H_{k + 1}} \right)} - H_{k - 1} - H_{k + 2} + 8}{16} \right\rbrack}}\quad$${k = {\frac{n}{2} + 2}},\ldots \quad,{n - 3}$$X_{n - 2} = {L_{\frac{n}{2} - 1} - \left\lbrack \frac{{2H_{\frac{n}{2} - 2}} + {2H_{\frac{n}{2} - 1}} - H_{\frac{n}{2}} + 4}{8} \right\rbrack}$$X_{n + 1} = {L_{\frac{n}{2}} - \left\lbrack \frac{{2H_{\frac{n}{2} + 1}} + {2H_{\frac{n}{2}}} - H_{\frac{n}{2} - 1} + 4}{8} \right\rbrack}$$X_{n + 3} = {L_{\frac{n}{2} + 1} - \left\lbrack \frac{H_{\frac{n}{2} + 1} + H_{\frac{n}{2} + 2} + 2}{4} \right\rbrack}$$X_{{2n} - 3} = {L_{n - 2} - \left\lbrack \frac{H_{n - 2} + H_{n - 1} + 2}{4} \right\rbrack}$$X_{{2n} - 1} = {L_{n - 1} - \left\lbrack \frac{H_{n - 1} + 2}{4} \right\rbrack}$$X_{1} = {H_{0} + \left\lbrack \frac{X_{0} + X_{2} + 1}{2} \right\rbrack}$${X_{{2k} + 1} = {H_{k} + \left\lbrack \frac{{9\left( {X_{2k} + X_{{2k} + 2}} \right)} - X_{{2k} - 2} - X_{{2k} + 4} + 8}{16} \right\rbrack}}\quad$${k = 1},\ldots \quad,{\frac{n}{2} - 3}$$X_{n - 3} = {H_{\frac{n}{2} - 2} + \left\lbrack \frac{X_{n - 4} + X_{n - 2} + 1}{2} \right\rbrack}$$X_{n - 1} = {H_{\frac{n}{2} - 1} + \left\lbrack \frac{{11X_{n - 2}} + {5X_{n + 1}} + 8}{16} \right\rbrack}$$X_{n} = {H_{\frac{n}{2}} + \left\lbrack \frac{{5X_{n - 2}} + {11X_{n + 1}} + 8}{16} \right\rbrack}$$X_{n + 2} = {H_{\frac{n}{2} + 1} + \left\lbrack \frac{X_{n + 1} + X_{n + 3} + 1}{2} \right\rbrack}$${X_{2k} = {H_{k} + \left\lbrack \frac{{9\left( {X_{{2k} - 1} + X_{{2k} + 1}} \right)} - X_{{2k} - 3} - X_{{2k} + 3} + 8}{16} \right\rbrack}}\quad$${k = {\frac{n}{2} + 2}},\ldots \quad,{n - 2}$${X_{{2n} - 2} = {H_{n - 1} + \left\lbrack \frac{X_{{2n} - 3} + X_{{2n} - 1} + 1}{2} \right\rbrack}}\quad$

When n is equal to eight, the above general set of inverse transformequations is reduced to the following:$X_{0} = {L_{0} - \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}$$X_{2} = {L_{1} - \left\lbrack \frac{H_{0} + H_{1} + 2}{4} \right\rbrack}$$X_{4} = {L_{2} - \left\lbrack \frac{H_{1} + H_{2} + 2}{4} \right\rbrack}$$X_{6} = {L_{3} - \left\lbrack \frac{{2H_{2}} + {2H_{3}} - H_{4} + 4}{8} \right\rbrack}$$X_{9} = {L_{4} - \left\lbrack \frac{{2H_{5}} + {2H_{4}} - H_{3} + 4}{8} \right\rbrack}$$X_{11} = {L_{5} - \left\lbrack \frac{H_{5} + H_{6} + 2}{4} \right\rbrack}$$X_{13} = {L_{6} - \left\lbrack \frac{H_{6} + H_{7} + 2}{4} \right\rbrack}$$X_{15} = {L_{7} - \left\lbrack \frac{H_{7} + 2}{4} \right\rbrack}$$X_{1} = {H_{0} + \left\lbrack \frac{X_{0} + X_{2} + 1}{2\quad} \right\rbrack}$$X_{3} = {H_{1} + \left\lbrack \frac{{9\left( {X_{2} + X_{4}} \right)} - X_{0} - X_{6} + 8}{16} \right\rbrack}$$X_{5} = {H_{2} + \left\lbrack \frac{X_{4} + X_{6} + 1}{2} \right\rbrack}$$X_{7} = {H_{3} + \left\lbrack \frac{{11X_{6}} + {5X_{9}} + 8}{16} \right\rbrack}$$X_{8} = {H_{4} + \left\lbrack \frac{{5X_{6}} + {11X_{9}} + 8}{16} \right\rbrack}$$X_{10} = {H_{5} + \left\lbrack \frac{X_{9} + X_{11} + 1}{2} \right\rbrack}$$X_{12} = {H_{6} + \left\lbrack \frac{{9\left( {X_{11} + X_{13}} \right)} - X_{9} - X_{15} + 8}{16} \right\rbrack}$$X_{14} = {H_{7} + \left\lbrack \frac{X_{13} + X_{15} + 1}{2} \right\rbrack}$

When n is equal to four, the inverse transform to be performed can berepresented as:

(L₀, L₁, L₂, L₃; H₀, H₁, H₂, H₃)→(X₀, X₁, X₂, X₃, X₄, X₅, X₆, X₇)

and the above general set of inverse transform equations is reduced tothe following: $\begin{matrix}{X_{0} = {L_{0} - \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}} \\{X_{2} = {L_{1} - \left\lbrack \frac{{2H_{0}} + {2H_{1}} - H_{2} + 4}{8} \right\rbrack}} \\{X_{5} = {L_{2} - \left\lbrack \frac{{2H_{3}} + {2H_{2}} - H_{1} + 4}{8} \right\rbrack}} \\{X_{7} = {L_{3} - \left\lbrack \frac{H_{3} + 2}{4} \right\rbrack}} \\{X_{1} = {H_{0} + \left\lbrack \frac{X_{0} + X_{2} + 1}{2} \right\rbrack}} \\{X_{3} = {H_{1} + \left\lbrack \frac{{11X_{2}} + {5X_{5}} + 8}{16} \right\rbrack}} \\{X_{4} = {H_{2} + \left\lbrack \frac{{5X_{2}} + {11X_{5}} + 8}{16} \right\rbrack}} \\{X_{6} = {H_{3} + \left\lbrack \frac{X_{5} + X_{7} + 1}{2} \right\rbrack}}\end{matrix}$

When n is equal to two, the inverse transform to be performed can berepresented as:

L₀, L₁; H₀, H₁)→(X₀, X₁, X₂, X₃, X₄)

and the above general set of inverse transform equations is reduced tothe following: $\begin{matrix}{X_{0} = {L_{0} - \left\lbrack \frac{H_{0} + 2}{4} \right\rbrack}} \\{X_{3} = {L_{1} - \left\lbrack \frac{H_{1} + 2}{4} \right\rbrack}} \\{X_{1} = {H_{0} + \left\lbrack \frac{X_{0} + X_{3} + 1}{2} \right\rbrack}} \\{X_{2} = {H_{1} + \left\lbrack \frac{X_{0} + X_{3} + 1}{2} \right\rbrack}}\end{matrix}$

In the preferred embodiment, during each layer of the inverse transformprocess the coefficients at the even positions (i.e., the X_(2i) values)must be computed before the coefficients at the odd positions (i.e., theX_(2i+1) values).

In an alternate embodiment, the short T1 decomposition transform is usedto filter all data, not just the data at the edges. Using only the shortT1 decomposition transform reduces computation time and complexity, butdecreases the data compression achieved and thus results in larger imagefiles. Using only the short transform also reduces the computation timeto decode an image file that contains an image encoded using the presentinvention, because only the corresponding short T1 reconstructiontransform is used during image reconstruction.

Adaptive Blockwise Quantization

Referring to FIG. 3, each wavelet coefficient produced by thewavelet-like decomposition transform is quantized:${\hat{x}}_{q} = {{{sign}(x)}\left\lbrack \left( {\frac{x}{q} + \frac{3}{8}} \right) \right\rbrack}$

where q is the quantization divisor, and is dequantized:

 {circumflex over (x)}=q{circumflex over (x)} _(q).

In accordance with the present invention, a quantization table is usedto assign each subband of the wavelet coefficients a quantizationdivisor, and thus controls the compression quality. If five layers ofwavelet transforms are performed for luminance values (and four layersfor the chrominance values), there are 16 subbands in the decompositionfor the luminance values:

LL₅, HL₅, LH₅, HH₅, HL₄, LH₄, HH₄, HL₃, LH₃, HH₃, HL₂, LH₂, HH₂, HL₁,LH₁, HH₁,

and 13 subbands for the chrominance values:

LL₄, HL₄, LH₄, HH₄, HL₃, LH₃, HH₃, HL₂, LH₂, HH₂, HL₁, LH₁, HH₁.

One possible quantization table for the luminance values is:

q=(16, 16, 16, 18, 18, 18, 24, 24, 24, 36, 46, 46, 93, 300, 300, 600)

and for the chrominance values:

q=(32, 50, 50, 100, 100, 100, 180, 200, 200, 400, 720, 720, 1440).

However, in a preferred embodiment, the quantization factor q is chosenadaptively for each distinct tile of the image, based on the density ofimage features in the tile. Referring to FIG. 8, we label the entries ofsubbands LH_(k), HL_(k) and HH_(k) as u_(ij) ^((k)), v_(ij) ^((k)), andw_(ij) ^((k)), respectively.

Referring to FIG. 9, the block classifier module computes for eachtransform layer (e.g., k=1, 2, 3, 4, 5) of the tile a set of blockclassification values, as follows: $\begin{matrix}{U_{k} = {\sum\limits_{i,j}\quad {u_{ij}^{(k)}}}} \\{V_{k} = {\sum\limits_{i,j}\quad {v_{ij}^{(k)}}}} \\{W_{k} = {\frac{1}{2}{\sum\limits_{i,j}\quad {w_{ij}^{(k)}}}}} \\{B_{k} = {\max \left\{ {U_{k},V_{k},W_{k}} \right\}}} \\{S_{k} = \sqrt{\frac{1}{2}\left\{ {U_{k}^{2} + V_{k}^{2} + W_{k}^{2} - {\frac{1}{3}\left( {U_{k} + V_{k} + W_{k}} \right)}} \right\}}}\end{matrix}$

Vertical and horizontal lines in the original image will mostly berepresented by u_(ij) ^((k)) and v_(ij) ^((k)), respectively. B_(k)tends to be large if the original image (i.e., in the tile beingevaluated by the block classifier) contains many features (e.g., edgesand textures). Therefore, the larger the value of B_(k), the harder itwill be to compress the image without creating compression artifacts.

Using a two-class model, two quantization tables are provided:

Q₀=(16, 16, 16, 18, 18, 18, 36, 36, 36, 72, 72, 72 144, 300, 300, 600),

Q₁=(16, 32, 32, 36, 36, 36, 72, 72, 72, 144, 144, 144, 288, 600, 600,1200)

where Q₀ is used for “hard” to compress blocks and Q₁ is used for “easy”to compress blocks.

Interior tiles (i.e., tiles not on the boundary of the image) are eachclassified as either “hard” or “easy” to compress based on a comparisonof one or more of the B_(k) values with one or more respective thresholdvalues. For instance, as shown in FIG. 9, B₁ for a tile may be comparedwith a first threshold TH1 (e.g., 65) (step 271). If B₁ is greater thanthe threshold, then the tile is classified as “hard” (step 272).Otherwise, B₅ is compared with a second threshold TH2 (e.g., 60) (step273). If B₅ is greater than the second threshold, then the tile isclassified as “hard” (step 274), and otherwise it is classified as“easy” (step 275). The wavelet coefficients for the tile are thenquantized using the quantization divisors specified by the quantizationtable corresponding to the block (i.e., tile) classification.

In the preferred embodiment, boundary tiles are classified by comparingB₁ with another, high threshold value TH1B, such as 85. Boundary tileswith a B₁ value above this threshold are classified as “hard” tocompress and otherwise are classified as “easy” to compress.

In an alternate embodiment, three or more block classifications may bedesignated, and a corresponding set of threshold values may be defined.Based on comparison of B₁ and/or other ones of the B₁ values with thesethresholds, a tile is classified into one of the designatedclassifications, and a corresponding quantization table is then selectedso as to determine the quantization values to be applied to the subbandswithin the tile. S_(k) also tends to be large if the original imagecontains many features, and therefore in some embodiments S_(k) is usedinstead of B_(k) to classify image tiles.

Sparse Data Encoding with Division between Significant and InsignificantPortions

Referring to FIGS. 10A and 10B, once the transform coefficients for atile of a base image have been generated and quantized, the next step isto encode the resulting coefficients of the tile. A group ofcomputational steps 280 are repeated for each NQS subband. Thebitstreams generated by encoding each NQS subband are divided by bitplanes and then grouped together to form the bitstreams stored in theimage files, as discussed above with respect to FIGS. 5A to 5E.

MaxbitDepth Mask

Referring to FIG. 10A, the encoding procedure or apparatus determinesthe maximum bit depth of the block of data in the NQS subband to beencoded (286), which is the maximum number of bits required to encodeany of the coefficient values in the block, and is herein called themaximum bit depth, or MaxbitDepth. The value of MaxbitDepth isdetermined by computing the maximum number of bits required to encodethe absolute value of any data value in the block. In particular,MaxbitDepth is equal to int(log2V)+1, where V is the largest absolutevalue of any element in the block, and “int( )” represents the integerportion of a specified value. The maximum bit depth for each top levelblock is stored in a corresponding bitstream (e.g., the significantbitstream for the subband group whose coefficients are being encoded).Next, the Block procedure is invoked for the current block (288). Apseudocode representation of the Block procedure is shown in Table 2.

Each block contains four subblocks (see FIG. 11A). As shown in FIG. 10B,the Block procedure determines the MaxbitDepth for each of the foursubblocks of the current block (300). Then, it generates and encodes aMaxbitDepth mask (301). The mask has four bits: m₁, m₂, m₃ and m₄, eachof which is set equal to a predefined value (e.g., 1) only if theMaxbitDepth of the corresponding subblock is equal to the MaxbitDepth m₀of the current (parent) block, and is otherwise set to zero. Themathematical representation of the mask is as follows:

mask=(m₀==m₁)+(m₀==m₂)+(M₀==m₃)+(M₀==m₄)

where the “+” in the above equation represents concatenation.

For example, a mask of 1000 indicates that only subblock 1 has aMaxbitDepth equal to the MaxbitDepth of the current block. The value ofthe mask is always between 1 and 15.

The MaxbitDepth mask is preferably encoded using a 15-symbol Huffmnantable (see Table 1). As can be seen, the four mask values thatcorrespond to the most common mask patterns, where just one subblockhaving a MaxbitDepth equal to the MaxbitDepth of the parent block, areencoded with just three bits.

TABLE 1 Huffman Table for Encoding MaxbitDepth Mask Mask Huffman Code0001  111 0010  101 0011  1001 0100  011 0101  0010 0110 10000 011101001 1000  110 1001 01000 1010  0001 1011 00110 1100  0101 1101 001111110  0000 1111 10001

Encoding Subblock MaxbitDepth Values

In addition, step 301 includes encoding the MaxbitDepth value for eachof the subblocks whose MaxbitDepth is not equal to the MaxbitDepth m₀ ofthe current block. For instance, as shown in FIGS. 11A and 11B, if theMaxbitDepth values for the current block are

m₁, m₂, m₃, m₄=5, 0, 3, 2

then the only MaxbitDepth values that need to be encoded are m₂, m₃ andm₄, because the MaxbitDepth value of m₁ is known from the MaxbitDepthmask and the previous stored and encoded value of the MaxbitDepth m₀ ofthe current block.

It should be noted that if m₀=1, then there is no need to encode theMaxbitDepth values of the subblocks, because those values are knowncompletely from the MaxbitDepth mask.

If m₀≠1, then for each m_(i)≠m₀, the procedure encodes the value m_(i)as follows:

m_(i)=0, then the procedure outputs a string of 0's of length m₀−1; and

otherwise, the procedure outputs a string of 0's of length m₀−m_(i)−1followed by a 1.

For instance, if m₀=5, and m₁=0, then m₁ is encoded as a string of four0's: 0000. If m₀=5, and m₂=3, then m₂ is encoded as a string of(5−3−1=1) one 0 followed by a 1:01.

In our example of {m₁, m₂, m₃, M₄}={5, 0, 3, 2}, the MaxbitDepth valuesare encoded as follows:

mask m₂ Subblock m₃ Subblock m₄ Subblock 11 0000 01 001

Next, if the coefficients of the NQS subband being encoded are to bestored in two or more bitstreams, then the encoded representation of theMaxbitDepth values for the block is divided into two more portions, witheach portion containing the information content for a certain range ofbit planes. For ease of explanation, we will explain in detail how theMaxbitDepth values and mask and coefficient values are split between twoportions, herein called the significant and insignificant portions. Thesame technique is used to split these values between three bit planeranges corresponding significant, mid-significant and insignificant (orleast significant) portions.

For each NQS subband, excluding the last group of NQS subbands, thecoefficient bit planes are divided into two or three ranges. When thereare two bit plane ranges, a bit plane threshold that divided the tworanges is chosen or predefined. The “insignificant” portion of each“coefficient value” (including its MaxbitDepth value) below the bitplane threshold is stored in an “insignificant” bitstream 206 (see FIG.5D), and the rest of the coefficient is stored in the correspondingsignificant bitstream 206. Selection of the bit plane ranges istypically done on an experimental basis, but encoding numerous imagesusing various bit plane ranges, and then selecting a set of bit planeranges that on average achieves a specified division of data between thebitstreams for the various resolution levels. For example, the specifieddivision may be an approximately equal division of data between thebitstream for a first resolution level and the next resolution level.Alternately, the specified division may call for the bitstreams for asecond resolution level to contain four times as much data as thebitstreams for a first (lower) resolution level.

We will first address the splitting of MaxbitDepth values betweensignificant and insignificant portions, and then we will address theencoding and splitting of coefficient values for minimum size blocks.

If the MaxbitDepth m₀ of a block is less than the threshold, theMaxbitDepth mask and every bit of the MaxbitDepth values for thesubblocks are stored in the insignificant portion of the base imagesubfile. Otherwise, the MaxbitDepth mask is stored in the significantpart, and then each of the encoded subblock MaxbitDepth values are splitbetween significant and insignificant parts as follows. This splittingis handled as follows. If m_(i)≧threshold, the entire encodedMaxbitDepth value m_(i) is included in the significant portion of thesubimage subfile. Otherwise, the first m₀—threshold bits of eachMaxbitDepth value m_(i), excluding m_(i)=m₀, are stored in thesignificant portion of the subimage subfile and the remaining bits ofeach m_(i) (if any) are stored in the insignificant portion of thesubimage subfile.

If the bit planes of the coefficients are to be divided into threeranges, then two bit plane thresholds are chosen or predefined, and theMaxbitDepth mask and values are allocated among three bitstreams usingthe same technique as described above.

Encoding Coefficient Values for Minimum Size Block

Next, if the size of the current block (i.e., the number of coefficientvalues in the current block) is not a predefined minimum number(302-No), such as four, then the Block procedure is called for each ofthe four subblocks of the current block (303). This is a recursiveprocedure call. As a result of calling the Block procedure on asubblock, the MaxbitDepth mask and values for the subblock are encodedand inserted into the pair of bitstreams for the subband group beingencoded. If the subblock is not of the predefined minimum size, then theBlock procedure is recursively called on its subblocks, and so on.

When a block of the predefined minimum size is processed by the blockprocedure (302-Yes), after the MaxbitDepth mask for the block and theMaxbitDepth values of the subblocks have been encoded (301), thecoefficients of the block are encoded, and the encoded values are splitbetween significant and insignificant parts (304).

Each coefficient that is not equal to zero includes a POS/NEG bit toindicate its sign, as well as a MaxbitDepth number of additional bits.Further, the MSB (most significant bit) of each non-zero coefficient,other than the sign bit, is already known from the MaxbitDepth value forthe coefficient, and in fact is known to be equal to 1. Therefore thisMSB does not need to be encoded (or from another viewpoint, it hasalready been encoded with the MaxbitDepth value).

For each coefficient of a minimum size block, if the MaxbitDepth of thecoefficient is less than the threshold, then all the bits of thecoefficient, including its sign bit, are in the insignificant portion.Otherwise, the sign bit is in the significant portion, and furthermorethe most significant bits (MSB's), if any, above the threshold number ofleast significant bits (LSB's), are also included in the significantportion. In other words, the bottom “threshold” number of bits areallocated to the insignificant portion. However, if the MaxbitDepth isequal to the threshold, the sign bit is nevertheless allocated to thesignificant portion and the remaining bits are allocated to theinsignificant portion.

Furthermore, as noted above, since the MSB of the absolute value of eachcoefficient is already known from the MaxbitDepth mask and values, thatbit is not stored. Also, coefficients with a value of zero are notencoded because their value is fully known from the MaxbitDepth value ofthe coefficient, which is zero.

For example (see FIG. 11C), consider four coefficients {31, 0, −5, −2}of a block whose values are with binary values are POS 11111, 0, NEG101, NEG 10, and a threshold value of 3. First the zero valuecoefficients and the MSB's of the non-zero coefficient are eliminated toyield: POS 1111, NEG 01, NEG 0. Then the threshold number of leastsignificant bits (other than sign bits) are allocated to theinsignificant portion and the rest are allocated to the significantportion as follows:

significant portion: POS 1, NEG

insignificant portion: 111, 01, NEG 0.

The significant portion contains the most significant bits of the 31 and−5 coefficient values, while the insignificant portion contains theremaining bits of the 31 and −5 coefficient values and all the bits ofthe −2 coefficient value.

TABLE 2 Pseudocode for Block Encoding Procedure // Encode MaxbitDepthm_(i) for each subblock i: Determine MaxbitDepth m_(i) for each subblocki = 1, 2, 3, 4 mask = (m₀ == m_(i)) + (m₀ == m₂) + (m₀ = m₃) + (m₀ ==m₄)  // wher the “+” in the above equation represents concatenationEncode and store mask using Huffman table For i=1 to 4 { If m_(i) ≠ m₀ {if m_(i) = 0 { output a string of m₀ 0's } else {  // m_(i) ≠ 0 output astring of m₀-m_(i) 0's, followed by a 1 } } } // Divide the encodedMaxbitDepth mask and MaxbitDepth between significant and //insignificant portions as follows: If m₀ < threshold { output theMaxbitDepth mask and MaxbitDepth values to insignificant bitstream }else { output the MaxbitDepth mask to significant bitstream; for i = 1to 4 { if m_(i) = m₀ {output nothing for that m_(i) } else { if m_(i) ≧threshold { output m_(i) to significant bitstream } else { output thefirst m₀-threshold bits of m_(i) to the significant bitstream and outputthe remaining bits of m_(i) (if any) in the insignificant bitstream } }} } // Encode Coefficient values if block is of minimum size If size ofcurrent block is > minimum block size { // coefficient values aredenoted as c_(i) for i = 1 to 4 { Call Block(subblock i); } else {  //size of current block is ≦minimum block size C = number of coefficientsin block; // if block size is already known, skip this step for i = 1 toC { if m_(i) < threshold { output all bits of c_(i) to insignificantbitstream; } else { output sign(c_(i)) to the significant bitstream; ifm_(i) > threshold { #M = m_(i) − threshold −1; // #M ≧ 0 output the #Mmost significant bits to the significant bitstream; } output allremaining least significant bits of c_(i) to the insignificantbitstream; } } // end of coefficient processing loop } // end of mainelse clause } // end of procedure Return

As discussed above, if the bit planes of the coefficients are to bedivided into three ranges, then two bit plane thresholds are chosen orpredefined, and the encoded coefficient values are allocated among threebitstreams using the same technique as described above.

Image Reconstruction

To reconstruct an image from an image file, at a specified resolutionlevel that is equal to or lower than the resolution level at which thebase image in the file was encoded, each bitstream of the image file upto the specified resolution level is decompressed and de-quantized.Then, on a tile by tile basis the reconstructed transform coefficientsare inverse transformed to reconstruct the image data at specifiedresolution level.

Referring to FIG. 12, the image reconstruction process reconstructs animage from image data received from an image file (320). A user of theprocedure or device performing the image reconstruction, or a controlprocedure operating on behalf of a user, selects or specifies aresolution level R that is equal to or less than the highest resolutionlevel included in the image data (322). A header of the image data fileis read to determine the number and arrangement of tiles (L, K) in theimage, and other information that may be needed by the imagereconstruction procedure (323). Steps 324 and 326 reconstruct the imageat the given resolution level, and at step 328 the reconstructed imageis displayed or stored in a memory device. FIGS. 13A and 13B provide amore detailed view of the procedure for decoding the data for aparticular tile at a particular subimage level.

In one preferred embodiment, as shown in FIG. 12, the data in the imagefile relevant to the specified resolution level is initially reorganizedinto tile by tile subfiles, with each tile subfile containing thebitstreams for that tile (324). Then, the data for each tile isprocessed (326). The header information is read to determine theMaxbitDepth for each top level subband block of the tile, thequantization factor used to quantize each subimage subband, and thelike. The transform coefficients for each NQS subband required toreconstruct the image at the specified resolution level are decoded, insubband order. The details of the decoding process for decoding thecoefficients in any one NQS subband are discussed below with referenceto FIG. 13B. The resulting decoded coefficients are de-quantizedapplying the quantization factors for each subband (obtained from the Qtable identified in the base image header). Then an inverse transform isapplied to the resulting de-quantized coefficients. Note that thewavelet-like inverse transforms for reconstructing an image from thedequantized transform coefficients have been described above.

Referring to FIG. 13A, to decode the data for one tile t at a specifiedresolution level, a set of steps 340 are repeated to decode each NQSsubband of the tile, excluding those NQS subbands not needed for thespecified resolution level and also excluding any bitstreams containingbit planes of encoded coefficient values not needed for the specifiedresolution level. Referring to FIGS. 5D and 5E, only the bitstreams ofthe base image needed to the specified resolution level are decoded. Fora particular top level block (corresponding to a NQS subband) of thetile being decoded, the MaxbitDepth of the top level block is determinedfrom either the header of the tile array (if the data has beenreorganized into tile arrays) or from the data at the beginning of thebitstream(s) for the subband (346), and then the Decode-Block procedureis called to decode the data for the current block (348).

After the data for a particular subband has been decoded, the decodedtransform coefficients for that subband may be de-quantized, applyingthe respective quantization factor for the respective (350).Alternately, de-quantization can be performed after all coefficients forall the subbands have been decoded.

Once all the coefficients for the NQS subbands have been decoded andde-quantized, an inverse transform is performed so as to regenerate theimage data for the current tile t at the specified resolution level(352).

In an alternate embodiment, step 324 of FIG. 12 is not used and the datain the image file is not reorganized into tile arrays. Rather, the imagedata is processed on a subband group by subband group basis, requiringthe recovered transform coefficients for all the tiles to be accumulatedand stored during the initial reconstruction steps. The steps 340 fordecoding the data for one top level block of a particular tile for aparticular subband group are repeated for each tile. In particular, fora particular top level block of a particular tile of a particularsubband group, the MaxbitDepth of the top level block is determined fromeither the header of the tile array or from the data at the beginning ofthe bitstream(s) for the subband group (346), and then the Decode-Blockprocedure is called to decode the data for the current block (348).

Referring to FIG. 13B, the Decode-Block procedure (which is applicableto both the preferred and alternate embodiments mentioned in thepreceding paragraphs) begins by decoding the MaxbitDepth data in theapplicable encoded data array so as to determine the MaxbitDepth of eachsubblock of the current block (360). Depending on the NQS subband beingdecoded, the MaxbitDepth data for a block may be in one bitstream or maybe split between two or three bitstreams, as described above, andtherefore the applicable MaxbitDepth data bits from all requiredbitstreams will be read and decoded. If the size of the current block isgreater than a predefined minimum block size (362-No), then theDecode-Block procedure is called for each of the subblocks of thecurrent block (363). This is a recursive procedure call. As a result ofcalling the Decode-Block procedure on a subblock, the MaxbitDepth valuesfor the subblock are decoded. If that subblock is not of the predefinedminimum size, then the Decode-Block procedure is recursively called onits subblocks, and so on.

When a block of the predefined minimum size is processed by theDecode-Block procedure (362-Yes), the coefficients of the block aredecoded. Depending on the subband group being decoded, the encodedcoefficients for a block may be in one bitstream or may be split betweentwo or three bitstreams, as described above, and therefore theapplicable data bits from all required bitstreams will be read anddecoded.

Referring to FIG. 13A, the quantized transform coefficients for eachtile are regenerated for all NQS subbands included in the specifiedresolution level. After these coefficients have been de-quantized, theinverse transform is applied to each tile (352), as already described.

Embodiment Using Non-Alternating Horizontal and Vertical Transforms

In another preferred embodiment, each tile of the image is firstprocessed by multiple (e.g., five) horizontal decomposition transformlayers and then by a similar number of vertical decomposition transformlayers. Equivalently, the vertical transform layers could be appliedbefore the horizontal transform layers. In hardware implementations ofthe image transformation methodology of the present invention, thischange in the order of the transform layers has the advantage of either(A) reducing the number of times the data array is rotated, or (B)avoiding the need for circuitry that switches the roles of rows andcolumns in the working image array(s). When performing successivehorizontal transforms, the second horizontal transform is applied to theleftmost array of low frequency coefficients generated by the firsthorizontal transform, and the third horizontal transform is applied tothe leftmost array of low frequency coefficients generated by the secondhorizontal transform, and so on. Thus, the second through Nth horizontaltransforms are applied to twice as much data as in the transform methodin which the horizontal and vertical transforms alternate. However, thisextra data processing generally does not take any additional processingtime in hardware implementations because in such implementations thehorizontal filter is applied simultaneously to all rows of the workingimage array.

The vertical transforms are applied in succession to successivelysmaller subarrays of the working image array. After the image data hasbeen transformed by all the transform layers (both horizontal andvertical) the quantization and encoding steps described above areapplied to the resulting transform coefficients to complete the imageencoding process.

As explained above, different (and typically shorter) transform filtersmay be applied to coefficients near the edges of the arrays beingprocessed than the (typically longer) transform filter applied tocoefficients away from those array edges. The use of longer transformfilters in the middle provides better data compression than the shortertransform filters, while the shorter transform filters eliminate theneed for data and coefficients from neighboring tiles.

Digital Camera Architecture

Referring to FIG. 14, there is shown an embodiment of a digital camerasystem 400 in accordance with the present invention. The digital camerasystem 400 includes an image capture device 402, such as a CCD sensorarray or any other mechanism suitable for capturing an image as an arrayof digitally encoded information. The image capture device is assumed toinclude analog to digital conversion (ADC) circuitry for convertinganalog image information into digital values.

A working memory 404, typically random access memory, receives digitallyencoded image information from the image capture device 402. Moregenerally, it is used to store a digitally encoded image while the imageis being transformed and compressed and otherwise processed by thecamera's data (i.e., image) processing circuitry 406. The dataprocessing circuitry 406 in one embodiment consists of hardwired logicand a set of state machines for performed a set of predefined imageprocessing operations.

In alternate embodiments the data processing circuitry 406 could beimplemented in part or entirely using a fast general purposemicroprocessor and a set of software procedures. However, at least usingthe technology available in 2000, it would be difficult to process andstore full resolution images (e.g., full color images having 1280×840pixels) fast enough to enable the camera to be able to take, say, 20pictures per second, which is a requirement for some commercialproducts. If, through the use of parallel processing techniques or welldesigned software, a low power, general purpose image datamicroprocessor could support the fast image processing needed by digitalcameras, then the data processing circuit 106 could be implemented usingsuch a general purpose microprocessor.

Each image, after it has been processed by the data processing circuitry406,is typically stored as an “image file” in a nonvolatile memorystorage device 408, typically implemented using “flash” (i.e., EEPROM)memory technology. The nonvolatile memory storage device 408 ispreferably implemented as a removable memory card. This allows thecamera's user to remove one memory card, plug in another, and then takeadditional pictures. However, in some implementations, the nonvolatilememory storage device 408 may not be removable, in which case the camerawill typically have a data access port 410 to enable the camera totransfer image files to and from other devices, such as general purpose,desktop computers. Digital cameras with removable nonvolatile memory 408may also include a data access port 410.

The digital camera 400 includes a set of buttons 412 for giving commandsto the camera. In addition to the image capture button, there willtypically be several other buttons to enable the use to select thequality level of the next picture to be taken, to scroll through theimages in memory for viewing on the camera's image viewer 414, to deleteimages from the nonvolatile image memory 408, and to invoke all thecamera's other functions. Such other functions might include enablingthe use of a flash light source, and transferring image files to andfrom a computer. The buttons in one embodiment are electromechanicalcontact switches, but in other embodiments at least some of the buttonsmay be implemented as touch screen buttons on a user interface display416, or on the image viewer 414.

The user interface display 416 is typically implemented either (A) as anLCD display device separate from the image viewer 414, or (B) as imagesdisplayed on the image viewer 414.

Menus, user prompts, and information about the images stored in thenonvolatile image memory 108 may be displayed on the user interfacedisplay 416, regardless of how that display is implemented.

After an image has been captured, processed and stored in nonvolatileimage memory 408, the associated image file may be retrieved from thememory 408 for viewing on the image viewer. More specifically, the imagefile is converted from its transformed, compressed form back into a dataarray suitable for storage in a framebuffer 418. The image data in theframebuffer is displayed on the image viewer 414. A date/time circuit420 is used to keep track of the current date and time, and each storedimage is date stamped with the date and time that the image was taken.

Still referring to FIG. 14, the digital camera 400 preferably includesdata processing circuitry 406 for performing a predefined set ofprimitive operations, such as performing the multiply and additionoperations required to apply a transform to a certain amount of imagedata, as well as a set of state machines 430-442 for controlling thedata processing circuitry so as to perform a set of predefined imagehandling operations. In one embodiment, the state machines in thedigital camera are as follows:

One or more state machines 430 for transforming, compressing and storingan image received from the camera's image capture mechanism. This imageis sometimes called the “viewfinder” image, since the image beingprocessed is generally the one seen on the camera's image viewer 414.This set of state machines 430 are the ones that initially generate eachimage file stored in the nonvolatile image memory 408. Prior to takingthe picture, the user specifies the quality level of the image to bestored, using the camera's buttons 412. In a preferred embodiment, theimage encoding state machines 430 implement one or more features of thepresent invention.

One or more state machines 432 for decompressing, inverse transformingand displaying a stored image file on the camera's image viewer. Thereconstructed image generated by decompressing, inverse transforming anddequantizing the image data is stored in camera's framebuffer 418 sothat it can be viewed on the image viewer 414.

One or more state machines 434 for updating and displaying a count ofthe number of images stored in the nonvolatile image memory 408. Theimage count is preferably displayed on the user interface display 416.This set of state machines 434 will also typically indicate whatpercentage of the nonvolatile image memory 408 remains unoccupied byimage files, or some other indication of the camera's ability to storeadditional images. If the camera does not have a separate interfacedisplay 416, this memory status information may be shown on the imageviewer 414, for instance superimposed on the image shown in the imageviewer 414 or shown in a region of the viewer 414 separate from the mainviewer image.

One or more state machines 436 for implementing a “viewfinder” mode forthe camera in which the image currently “seen” by the image capturemechanism 402 is displayed on the image viewer 414 to that the user cansee the image that would be stored if the image capture button ispressed. These state machines transfer the image received from the imagecapture device 402, possibly after appropriate remedial processing stepsare performed to improve the raw image data, to the camera's framebuffer418.

One or more state machines 438 for downloading images from thenonvolatile image memory 408 to an external device, such as a generalpurpose computer.

One or more state machines 440 for uploading images from an externaldevice, such as a general purpose computer, into the nonvolatile imagememory 408. This enables the camera to be used as an image viewingdevice, and also as a mechanism for transferring image files on memorycards.

Alternate Embodiments

Generally, the present invention is useful in any “memory conservative”context where the amount of working memory available is insufficient toprocess entire images as a single tile, or where a product must work ina variety of environments including low memory environments, or where animage may need to be conveyed over a low bandwidth communication channelor where it may be necessary or convenient to provide an image at avariety of resolution levels.

In streaming data implementations, such as in a web browser thatreceives compressed images encoded using the present invention,subimages of an image may be decoded and decompressed on the fly, as thedata for other higher level subimages of the image are being received.As a result, one or more lower resolution versions of the compressedimage may be reconstructed and displayed before the data for the highestresolution version of the image is received (and/or decoded) over acommunication channel.

In another alternate embodiment, a different transform than thewavelet-like transform described above could be used.

In alternate embodiments the image tiles could be processed in adifferent order. For instance, the image tiles could be processed fromright to left instead of left to right. Similarly, image tiles could beprocessed starting at the bottom row and proceeding toward the top row.

The present invention can be implemented as a computer program productthat includes a computer program mechanism embedded in a computerreadable storage medium. For instance, the computer program productcould contain the program modules shown in FIG. 2. These program modulesmay be stored on a CD-ROM, magnetic disk storage product, or any othercomputer readable data or program storage product. The software modulesin the computer program product may also be distributed electronically,via the Internet or otherwise, by transmission of a computer data signal(in which the software modules are embedded) on a carrier wave.

While the present invention has been described with reference to a fewspecific embodiments, the description is illustrative of the inventionand is not to be construed as limiting the invention. Variousmodifications may occur to those skilled in the art without departingfrom the true spirit and scope of the invention as defined by theappended claims.

What is claimed is:
 1. A method of processing an array of image data,comprising: applying a predefined family of transform filters to thearray of image data so as to generate successive sets of transformcoefficients, each set of transform coefficients including edgecoefficients at outside boundaries of the set of transform coefficientsand non-edge coefficients positioned at interior locations of the set oftransform coefficients; the sets of transform coefficients including alast set of transform coefficients and one or more earlier sets oftransform coefficients; and the applying including applying a shorttransform filter to image data at boundaries of the array and tocoefficients positioned at boundaries of each earlier set of transformcoefficients so as to generate the edge coefficients, and applying along filter to image data at interior locations of the array and tocoefficients at interior locations of the earlier sets of transformcoefficients so as to generate the non-edge coefficients; wherein theboundaries of the array and the interior locations of the array arepredetermined without respect to content of the array; wherein the shorttransform filter has a shorter filter support than the long transformfilter, and both the short transform filter and the long transformfilter are applied only to image data within the array and only totransform coefficients within the earlier sets of transform coefficientsfor the array while generating the sets of transform coefficients. 2.The method of claim 1, wherein the short transform filter includes ashort low spatial frequency filter that generates low spatial frequencycoefficients, and the short low spatial frequency filter weights theimage datum closest to the boundary of the array and the transformcoefficient closest to the boundary of each earlier set of transformcoefficients so as to as enable regeneration of the image from thetransform coefficients without array boundary artifacts.
 3. The methodof claim 2, wherein the short low spatial frequency filter weights thetransform coefficient closest to the boundary of one of the earlier setof transform coefficients so that more than 50% of the edge transformcoefficient generated by the filter for a next set of transformcoefficients is attributable to the transform coefficient closest to theboundary of one of the earlier sets of transform coefficients, said nextset of transform coefficients comprising one of said sets of transformcoefficients.
 4. The method of claim 1, wherein a plurality of thetransform filters are asymmetric boundary filters, extending to a firstextent to each array's boundary, and extending to a second, longer,extent in a direction away from the array's boundary, but not extendingover the array's boundary.
 5. The method of claim 1, wherein thetransform filters are wavelet or wavelet-like decomposition transformfilters.
 6. The method of claim 1, wherein the long filter includes acenter filter, applied to image data at a predefined set of arraylocations within the interior locations, for generating a set of centerhigh pass and low pass transform coefficients and the long filterfurthermore includes first and second distinct interior filters, whereinthe first interior filter is applied to array locations on a first sideof the predefined set of array locations and the second interior filteris applied to array locations on a distinct, second side of thepredefined set of array locations.
 7. A method of processing an array ofimage data, comprising: processing tiles of the image data in apredefined order, the tiles comprising nonoverlapping portions of theimage data, so as to generate processed image data; and storing theprocessed image data as a data image file; the processing of each tileof image data comprising: applying a predefined family of transformfilters to the tile of image data so as to generate successive sets oftransform coefficients, each set of transform coefficients includingedge coefficients positioned at outside boundaries of the set oftransform coefficients and non-edge coefficients positioned at interiorlocations of the set of transform coefficients; the sets of transformcoefficients including a last set of transform coefficients and one ormore earlier sets of transform coefficients; and the applying includingapplying a short transform filter to image data at boundaries of thetile and to coefficients positioned at boundaries of each earlier set oftransform coefficients so as to generate the edge coefficients, andapplying a long filter to image data at interior locations of the tileand to coefficients at interior locations of the earlier sets oftransform coefficients so as to generate the non-edge coefficients;wherein the boundaries of the array and the interior locations of thearray are predetermined without respect to content of the array; whereinthe short transform filter has a shorter filter support than the longtransform filter, and both the short transform filter and the longtransform filter are applied only to image data within the tile and onlyto transform coefficients within the earlier sets of transformcoefficients for the tile while generating the sets of transformcoefficients.
 8. The method of claim 7, wherein the processing of eachtile includes: for at least two of the respective sets of transformcoefficients, generating one or more parameters whose value isindicative of density of image features in the tile; classifying thetile into one of a predefined set of categories in accordance with thevalues of the one or more parameters; selecting a set of quantizationfactors for each respective tile in accordance with the category intowhich the tile has been classified; and scaling the transformcoefficients of the tile by the set of quantization factors to as togenerate a set of quantized transform coefficients for the tile.
 9. Themethod of claim 8, including storing in the image data file anindication of the selected set of quantization factors for each tile ofthe image data.
 10. A computer program product for use in conjunctionwith a computer system, the computer program product comprising acomputer readable storage medium and a computer program mechanismembedded therein, the computer program mechanism comprising: an imageprocessing module that applies a predefined family of transform filtersto the array of image data so as to generate successive sets oftransform coefficients, each set of transform coefficients includingedge coefficients positioned at outside boundaries of the set oftransform coefficients and non-edge coefficients positioned at interiorlocations of the set of transform coefficients; the sets of transformcoefficients including a last set of transform coefficients and one ormore earlier sets of transform coefficients; and the predefined familyof transform filters including a short transform filter applied by theimage processing module to image data at boundaries of the array and tocoefficients positioned at boundaries of each earlier set of transformcoefficients so as to generate the edge coefficients, and applying along filter applied by the image processing module to image data atinterior locations of the array and to coefficients at interiorlocations of the earlier sets of transform coefficients so as togenerate the non-edge coefficients; wherein the boundaries of the arrayand the interior locations of the array are predetermined withoutrespect to content of the array; wherein the short transform filter hasa shorter filter support than the long transform filter, and both theshort transform filter and the long transform filter are applied only toimage data within the array and only to transform coefficients withinthe earlier sets of transform coefficients for the array whilegenerating the sets of transform coefficients.
 11. The computer programproduct of claim 10, wherein the short transform filter includes a shortlow spatial frequency filter that generates low spatial frequencycoefficients, and the short low spatial frequency filter weights theimage datum closest to the boundary of the array and the transformcoefficient closest to the boundary of each earlier set of transformcoefficients so as to as enable regeneration of the image from thetransform coefficients without array boundary artifacts.
 12. Thecomputer program product of claim 11, wherein the short low spatialfrequency filter weights the transform coefficient closest to theboundary of one of the earlier set of transform coefficients so thatmore than 50% of the edge transform coefficient generated by the filterfor a next set of transform coefficients is attributable to thetransform coefficient closest to the boundary of one of the earlier setsof transform coefficients, said next set of transform coefficientscomprising one of said sets of transform coefficients.
 13. The computerprogram product of claim 10, wherein a plurality of the transformfilters are asymmetric boundary filters, extending to a first extent toeach array's boundary, and extending to a second, longer, extent in adirection away from the array's boundary, but not extending over thearray's boundary.
 14. The computer program product of claim 10, whereinthe transform filters are wavelet or wavelet-like decompositiontransform filters.
 15. The computer program product of claim 10, whereinthe interior filter includes a center filter, applied to image data at apredefined set of array locations within the interior locations, forgenerating a set of center high pass and low pass transform coefficientsand the interior filter furthermore includes first and second distinctinterior filters, wherein the first interior filter is applied to arraylocations on a first side of the predefined set of array locations andthe second interior filter is applied to array locations on a distinct,second side of the predefined set of array locations.
 16. A computerprogram product for use in conjunction with a computer system, thecomputer program product comprising a computer readable storage mediumand a computer program mechanism embedded therein, the computer programmechanism comprising: an image processing module that processes tiles ofthe image data in a predefined order, the tiles comprisingnonoverlapping portions of the image data, so as to generate processedimage data, and that stores the processed image data as a data imagefile; the image processing module including instructions for processingof each tile of image data, comprising: instructions for applying apredefined family of transform filters to the tile of image data, so asto generate successive sets of transform coefficients, each set oftransform coefficients including edge coefficients positioned at outsideboundaries of the set of transform coefficients and non-edgecoefficients positioned at interior locations of the set of transformcoefficients; the sets of transform coefficients including a last set oftransform coefficients and one or more earlier sets of transformcoefficients; and the instructions for applying a predefined family oftransform filters including instructions for applying a short transformfilter to image data at boundaries of the tile and to coefficientspositioned at boundaries of each earlier set of transform coefficientsso as to generate the edge coefficients, and for applying a long filterto image data at interior locations of the tile and to coefficients atinterior locations of the earlier sets of transform coefficients so asto generate the non-edge coefficients; wherein the boundaries of thearray and the interior locations of the array are predetermined withoutrespect to content of the array; wherein the short transform filter hasa shorter filter support than the long transform filter, and both theshort transform filter and the long transform filter are applied only toimage data within the tile and only to transform coefficients within theearlier sets of transform coefficients for the tile while generating thesets of transform coefficients.
 17. The computer program product ofclaim 16, wherein the image processing module includes: instructions forgenerating, for at least two of the respective sets of transformcoefficients, one or more parameters whose value is indicative ofdensity of image features in the tile; instructions for classifying thetile into one of a predefined set of categories in accordance with thevalues of the one or more parameters; instructions for selecting a setof quantization factors for each respective tile in accordance with thecategory into which the tile has been classified; and instructions forscaling the transform coefficients of the tile by the set ofquantization factors to as to generate a set of quantized transformcoefficients for the tile.
 18. The computer program product of claim 17,including instructions for storing in the image data file an indicationof the selected set of quantization factors for each tile of the imagedata.
 19. A method of processing an array of image data, comprising:applying a predefined family of transform filters to the array of imagedata so as to generate successive sets of transform coefficients,including applying a first family of transform filters to the array ofimage data so as to generate a first set of transform coefficients andapplying a second family of transform filters to the first set oftransform coefficients so as to generate a second set of transformcoefficients; wherein applying the first family of transform filtersincludes applying a first short transform filter to image data atboundaries of the array and a first long transform filter to image dataat interior locations of the array, and applying the second family oftransform filters includes applying a second short transform filter toedge coefficients at boundaries of the first set of transformcoefficients and a second long transform filter to coefficients atinterior locations of the first set of transform coefficients; whereinthe boundaries of the array and the first set of transform coefficients,and the interior locations of the array and the first set of transformcoefficients are predetermined without respect to content of the arrayand the first set of transform coefficients; wherein the first family oftransform filters have shorter filter supports than the second family oftransform filters.
 20. The method of claim 19, including applying thesecond family of transform filters to the second set of transformcoefficients so as to generate a third set of transform coefficients andapplying the second family of transform filters to the third set oftransform coefficients so as to generate a fourth set of transformcoefficients wherein applying the second family of transform filtersincludes applying the second short transform filter to edge coefficientsat boundaries of the second and the third set of transform coefficientsand applying the second long transform filter to coefficients atinterior locations of the second and the third set of transformcoefficients; wherein the boundaries of the second and the third set oftransform coefficients, and the interior locations of the second and thethird set of transform coefficients are predetermined without respect tocontent of the second and the third set of transform coefficients.
 21. Acomputer program product for use in conjunction with a computer system,the computer program product comprising a computer readable storagemedium and a computer program mechanism embedded therein, the computerprogram mechanism comprising: an image processing module that applies apredefined family of transform filters to the array of image data so asto generate successive sets of transform coefficients, each set oftransform coefficients including edge coefficients positioned at outsideboundaries of the set of transform coefficients and non-edgecoefficients positioned at interior locations of the set of transformcoefficients; the predefined family of transform filters including afirst family of transform filters applied by the image processing moduleto the array of image data so as to generate a first set of transformcoefficients and a second family of transform filters applied by theimage processing module to the first set of transform coefficients so asto generate a second set of transform coefficients; wherein the imageprocessing module applies a first short transform filter to image dataat boundaries of the array and a first long transform filter to imagedata at interior locations of the array, and the image processing moduleapplies a second short transform filter to edge coefficients atboundaries of the first set of transform coefficients and a second longtransform filter to coefficients at interior locations of the first setof transform coefficients; wherein the boundaries of the array and thefirst set of transform coefficients, and the interior locations of thearray and the first set of transform coefficients are predeterminedwithout respect to content of the array and the first set of transformcoefficients; wherein the first family of transform filters have shorterfilter supports than the second family of transform filters.
 22. Thecomputer program product of claim 21, wherein the image processingmodule includes instructions for applying the second family of transformfilters to the second set of transform coefficients so as to generate athird set of transform coefficients and for applying the second familyof transform filters to the third set of transform coefficients so as togenerate a fourth set of transform coefficients wherein applying thesecond family of transform filters includes applying the second shorttransform filter to edge coefficients at boundaries of the second andthe third set of transform coefficients and applying the second longtransform filter to coefficients at interior locations of the second andthe third set of transform coefficients; wherein the boundaries of thesecond and the third set of transform coefficients, and the interiorlocations of the second and the third set of transform coefficients arepredetermined without respect to content of the second and the third setof transform coefficients.